[jira] [Resolved] (KUDU-2401) External TLS certificate with Intermediate CA in server cert file fails
[ https://issues.apache.org/jira/browse/KUDU-2401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sailesh Mukil resolved KUDU-2401. - Resolution: Fixed Fix Version/s: 1.7.1 https://github.com/apache/kudu/commit/109fdc84e45523fdbd8b1ca29110965187b10323 > External TLS certificate with Intermediate CA in server cert file fails > --- > > Key: KUDU-2401 > URL: https://issues.apache.org/jira/browse/KUDU-2401 > Project: Kudu > Issue Type: Bug > Components: security >Reporter: Sailesh Mukil >Assignee: Sailesh Mukil >Priority: Major > Labels: security, tls > Fix For: 1.7.1 > > > This was found while using Impala w/ KRPC with external PKI. > Take 2 certificate files: cert.pem and truststore.pem > cert.pem has 2 certificates in it: > A cert for that node (with CN="hostname", and signed by CN=CertToolkitIntCA) > And the intermediate CA cert (with CN=CertToolkitIntCA, and signed by > CN=CertToolkitRootCA) > truststore.pem has 1 certificate in it: > A cert which is the root CA (with CN=CertToolkitRootCA, self-signed) > This format of certificates works with Impala on Thrift but it doesn't work > with KRPC. > Workaround for this issue w/ KRPC turned on: > If we move the second certificate from cert.pem (CN=CertToolkitIntCA) into > truststore.pem, then this seems to work. > Also TODO: Add a test case that has multiple intermediate CAs. Right now > we're testing with only one intermediate CA. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (KUDU-2092) Pull in krb5_is_config_principal() when running against older kerberos versions
[ https://issues.apache.org/jira/browse/KUDU-2092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sailesh Mukil reassigned KUDU-2092: --- Assignee: Lars Volker (was: Sailesh Mukil) > Pull in krb5_is_config_principal() when running against older kerberos > versions > --- > > Key: KUDU-2092 > URL: https://issues.apache.org/jira/browse/KUDU-2092 > Project: Kudu > Issue Type: Improvement > Components: security >Affects Versions: 1.4.0 >Reporter: Sailesh Mukil >Assignee: Lars Volker >Priority: Major > > On kerberos versions < krb5-1.8, the function krb5_is_config_principal() does > not exist. Since our code uses that function, and we dynamically link against > kerberos, we would be unable to build on systems that have these old kerberos > versions. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KUDU-2401) External TLS certificate with Intermediate CA in server cert file fails
[ https://issues.apache.org/jira/browse/KUDU-2401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sailesh Mukil updated KUDU-2401: Description: This was found while using Impala w/ KRPC with external PKI. Take 2 certificate files: cert.pem and truststore.pem cert.pem has 2 certificates in it: A cert for that node (with CN="hostname", and signed by CN=CertToolkitIntCA) And the intermediate CA cert (with CN=CertToolkitIntCA, and signed by CN=CertToolkitRootCA) truststore.pem has 1 certificate in it: A cert which is the root CA (with CN=CertToolkitRootCA, self-signed) This format of certificates works with Impala on Thrift but it doesn't work with KRPC. Workaround for this issue w/ KRPC turned on: If we move the second certificate from cert.pem (CN=CertToolkitIntCA) into truststore.pem, then this seems to work. Also TODO: Add a test case that has multiple intermediate CAs. Right now we're testing with only one intermediate CA. was: This was found while using Impala w/ KRPC with external PKI. Take 2 certificate files: cert.pem and truststore.pem cert.pem has 2 certificates in it: A cert for that node (with CN="hostname", and signed by CN=CertToolkitIntCA) And the intermediate CA cert (with CN=CertToolkitIntCA, and signed by CN=CertToolkitRootCA) truststore.pem has 1 certificate in it: A cert which is the root CA (with CN=CertToolkitRootCA, self-signed) This format of certificates works with Impala on Thrift but it doesn't work with KRPC. Workaround for this issue w/ KRPC turned on: If we move the second certificate from cert.pem (CN=CertToolkitIntCA) into truststore.pem, then this seems to work. > External TLS certificate with Intermediate CA in server cert file fails > --- > > Key: KUDU-2401 > URL: https://issues.apache.org/jira/browse/KUDU-2401 > Project: Kudu > Issue Type: Bug > Components: security >Reporter: Sailesh Mukil >Assignee: Sailesh Mukil >Priority: Major > Labels: security, tls > > This was found while using Impala w/ KRPC with external PKI. > Take 2 certificate files: cert.pem and truststore.pem > cert.pem has 2 certificates in it: > A cert for that node (with CN="hostname", and signed by CN=CertToolkitIntCA) > And the intermediate CA cert (with CN=CertToolkitIntCA, and signed by > CN=CertToolkitRootCA) > truststore.pem has 1 certificate in it: > A cert which is the root CA (with CN=CertToolkitRootCA, self-signed) > This format of certificates works with Impala on Thrift but it doesn't work > with KRPC. > Workaround for this issue w/ KRPC turned on: > If we move the second certificate from cert.pem (CN=CertToolkitIntCA) into > truststore.pem, then this seems to work. > Also TODO: Add a test case that has multiple intermediate CAs. Right now > we're testing with only one intermediate CA. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KUDU-2401) External TLS certificate with Intermediate CA in server cert file fails
[ https://issues.apache.org/jira/browse/KUDU-2401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16427635#comment-16427635 ] Sailesh Mukil commented on KUDU-2401: - Patch for review in: https://gerrit.cloudera.org/c/9934/ > External TLS certificate with Intermediate CA in server cert file fails > --- > > Key: KUDU-2401 > URL: https://issues.apache.org/jira/browse/KUDU-2401 > Project: Kudu > Issue Type: Bug > Components: security >Reporter: Sailesh Mukil >Assignee: Sailesh Mukil >Priority: Major > Labels: security, tls > > This was found while using Impala w/ KRPC with external PKI. > Take 2 certificate files: cert.pem and truststore.pem > cert.pem has 2 certificates in it: > A cert for that node (with CN="hostname", and signed by CN=CertToolkitIntCA) > And the intermediate CA cert (with CN=CertToolkitIntCA, and signed by > CN=CertToolkitRootCA) > truststore.pem has 1 certificate in it: > A cert which is the root CA (with CN=CertToolkitRootCA, self-signed) > This format of certificates works with Impala on Thrift but it doesn't work > with KRPC. > Workaround for this issue w/ KRPC turned on: > If we move the second certificate from cert.pem (CN=CertToolkitIntCA) into > truststore.pem, then this seems to work. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KUDU-2385) Kerberos ticket reacquisition with Heimdal krb5 likely broken
[ https://issues.apache.org/jira/browse/KUDU-2385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sailesh Mukil updated KUDU-2385: Description: [~kwho] found that when we call krb5_cc_store_cred(), it looks like we should be passing in 'new_creds' instead of 'creds': https://github.com/apache/kudu/blob/b357fa9b729ce52627a862e08ebac822ae470bc9/src/kudu/security/init.cc#L279 was: When we call krb5_cc_store_cred(), it looks like we should be passing in 'new_creds' instead of 'creds': https://github.com/apache/kudu/blob/b357fa9b729ce52627a862e08ebac822ae470bc9/src/kudu/security/init.cc#L279 > Kerberos ticket reacquisition with Heimdal krb5 likely broken > - > > Key: KUDU-2385 > URL: https://issues.apache.org/jira/browse/KUDU-2385 > Project: Kudu > Issue Type: Bug > Components: security >Reporter: Sailesh Mukil >Priority: Major > > [~kwho] found that when we call krb5_cc_store_cred(), it looks like we should > be passing in 'new_creds' instead of 'creds': > https://github.com/apache/kudu/blob/b357fa9b729ce52627a862e08ebac822ae470bc9/src/kudu/security/init.cc#L279 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KUDU-2385) Kerberos ticket reacquisition with Heimdal krb5 likely broken
Sailesh Mukil created KUDU-2385: --- Summary: Kerberos ticket reacquisition with Heimdal krb5 likely broken Key: KUDU-2385 URL: https://issues.apache.org/jira/browse/KUDU-2385 Project: Kudu Issue Type: Bug Components: security Reporter: Sailesh Mukil When we call krb5_cc_store_cred(), it looks like we should be passing in 'new_creds' instead of 'creds': https://github.com/apache/kudu/blob/b357fa9b729ce52627a862e08ebac822ae470bc9/src/kudu/security/init.cc#L279 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KUDU-2301) Add metrics per connection to the reactor metrics
[ https://issues.apache.org/jira/browse/KUDU-2301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16390346#comment-16390346 ] Sailesh Mukil commented on KUDU-2301: - Part 1 is committed here: https://github.com/apache/kudu/commit/68d797da1c11e15ebde06b16bb7e6ef3f36f9994 > Add metrics per connection to the reactor metrics > - > > Key: KUDU-2301 > URL: https://issues.apache.org/jira/browse/KUDU-2301 > Project: Kudu > Issue Type: Task > Components: rpc >Reporter: Sailesh Mukil >Assignee: Sailesh Mukil >Priority: Major > Labels: metrics, observability > > We can expose metrics on a per connection level and store them in > ReactorMetrics. > As an initial step, we can expose the OutboundTransfer queue size and a > rolling average of transfer speeds in Kbps. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KUDU-2301) Add metrics per connection to the reactor metrics
Sailesh Mukil created KUDU-2301: --- Summary: Add metrics per connection to the reactor metrics Key: KUDU-2301 URL: https://issues.apache.org/jira/browse/KUDU-2301 Project: Kudu Issue Type: Task Components: rpc Reporter: Sailesh Mukil Assignee: Sailesh Mukil We can expose metrics on a per connection level and store them in ReactorMetrics. As an initial step, we can expose the OutboundTransfer queue size and a rolling average of transfer speeds in Kbps. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KUDU-2228) Make Messenger options configurable
Sailesh Mukil created KUDU-2228: --- Summary: Make Messenger options configurable Key: KUDU-2228 URL: https://issues.apache.org/jira/browse/KUDU-2228 Project: Kudu Issue Type: Task Components: rpc Reporter: Sailesh Mukil Assignee: Sailesh Mukil Currently, the RPC layer accesses many gflags directly to take certain decisions, eg. whether to turn on encryption, authentication, etc. Since the RPC layer is to be used more like a library, these should be configurable options that are passed to the Messenger (which is the API endpoint for the application using the RPC layer), instead of the RPC layer itself directly accessing these flags. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KUDU-2228) Make Messenger options configurable
[ https://issues.apache.org/jira/browse/KUDU-2228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16273482#comment-16273482 ] Sailesh Mukil commented on KUDU-2228: - CC: [~kwho] > Make Messenger options configurable > --- > > Key: KUDU-2228 > URL: https://issues.apache.org/jira/browse/KUDU-2228 > Project: Kudu > Issue Type: Task > Components: rpc >Reporter: Sailesh Mukil >Assignee: Sailesh Mukil > Labels: refactor, rpc > > Currently, the RPC layer accesses many gflags directly to take certain > decisions, eg. whether to turn on encryption, authentication, etc. > Since the RPC layer is to be used more like a library, these should be > configurable options that are passed to the Messenger (which is the API > endpoint for the application using the RPC layer), instead of the RPC layer > itself directly accessing these flags. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KUDU-2220) GetEndOfChainX509 does not return end-user cert
[
https://issues.apache.org/jira/browse/KUDU-2220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16261326#comment-16261326
]
Sailesh Mukil commented on KUDU-2220:
-
Commit in:
https://github.com/apache/kudu/commit/3e59fd7b14b4a2ba2846621df04093cce9024688
> GetEndOfChainX509 does not return end-user cert
> ---
>
> Key: KUDU-2220
> URL: https://issues.apache.org/jira/browse/KUDU-2220
> Project: Kudu
> Issue Type: Bug
> Components: security
>Affects Versions: 1.5.0
>Reporter: Sailesh Mukil
>Assignee: Sailesh Mukil
> Fix For: 1.6.0
>
>
> KUDU-2091 introduced a function GetEndOfChainX509() which was supposed to
> return the "end-user" certificate. However, the end-user certificate is not
> at the end of the chain, but rather at the beginning of the chain as
> specificed by the RFC:
> https://tools.ietf.org/html/rfc5246#section-7.4.2
> {quote}This is a sequence (chain) of certificates. The sender's certificate
> MUST come first in the list. Each following certificate MUST directly
> certify the one preceding it.{quote}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (KUDU-2220) GetEndOfChainX509 does not return end-user cert
[
https://issues.apache.org/jira/browse/KUDU-2220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16259553#comment-16259553
]
Sailesh Mukil commented on KUDU-2220:
-
Patch out for review:
https://gerrit.cloudera.org/#/c/8595/
> GetEndOfChainX509 does not return end-user cert
> ---
>
> Key: KUDU-2220
> URL: https://issues.apache.org/jira/browse/KUDU-2220
> Project: Kudu
> Issue Type: Bug
> Components: security
>Affects Versions: 1.5.0
>Reporter: Sailesh Mukil
>Assignee: Sailesh Mukil
>
> KUDU-2091 introduced a function GetEndOfChainX509() which was supposed to
> return the "end-user" certificate. However, the end-user certificate is not
> at the end of the chain, but rather at the beginning of the chain as
> specificed by the RFC:
> https://tools.ietf.org/html/rfc5246#section-7.4.2
> {quote}This is a sequence (chain) of certificates. The sender's certificate
> MUST come first in the list. Each following certificate MUST directly
> certify the one preceding it.{quote}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (KUDU-2220) GetEndOfChainX509 does not return end-user cert
Sailesh Mukil created KUDU-2220:
---
Summary: GetEndOfChainX509 does not return end-user cert
Key: KUDU-2220
URL: https://issues.apache.org/jira/browse/KUDU-2220
Project: Kudu
Issue Type: Bug
Components: security
Affects Versions: 1.5.0
Reporter: Sailesh Mukil
Assignee: Sailesh Mukil
KUDU-2091 introduced a function GetEndOfChainX509() which was supposed to
return the "end-user" certificate. However, the end-user certificate is not at
the end of the chain, but rather at the beginning of the chain as specificed by
the RFC:
https://tools.ietf.org/html/rfc5246#section-7.4.2
{quote}This is a sequence (chain) of certificates. The sender's certificate
MUST come first in the list. Each following certificate MUST directly certify
the one preceding it.{quote}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (KUDU-2092) Pull in krb5_is_config_principal() when running against older kerberos versions
[ https://issues.apache.org/jira/browse/KUDU-2092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16203854#comment-16203854 ] Sailesh Mukil edited comment on KUDU-2092 at 10/13/17 5:00 PM: --- [~tlipcon] Yes, it is. I just realized that Henry added it in Impala's fetch of the Kudu security code in one of his patches, so we're diverging in code there. I can do that patch in Kudu as well to avoid this divergence, and when the time comes for us to rebase onto a newer version of KRPC and Kudu security, it should resolve itself. was (Author: sailesh): [~tlipcon] Yes, it is. I just realized that Henry added it in the Impala code in one of his patches, so we're diverging in code there. I can do that patch in Kudu as well to avoid this divergence, and when the time comes for us to rebase onto a newer version of KRPC and Kudu security, it should resolve itself. > Pull in krb5_is_config_principal() when running against older kerberos > versions > --- > > Key: KUDU-2092 > URL: https://issues.apache.org/jira/browse/KUDU-2092 > Project: Kudu > Issue Type: Improvement > Components: security >Affects Versions: 1.4.0 >Reporter: Sailesh Mukil >Assignee: Sailesh Mukil > > On kerberos versions < krb5-1.8, the function krb5_is_config_principal() does > not exist. Since our code uses that function, and we dynamically link against > kerberos, we would be unable to build on systems that have these old kerberos > versions. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KUDU-2092) Pull in krb5_is_config_principal() when running against older kerberos versions
[ https://issues.apache.org/jira/browse/KUDU-2092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16203854#comment-16203854 ] Sailesh Mukil commented on KUDU-2092: - [~tlipcon] Yes, it is. I just realized that Henry added it in the Impala code in one of his patches, so we're diverging in code there. I can do that patch in Kudu as well to avoid this divergence, and when the time comes for us to rebase onto a newer version of KRPC and Kudu security, it should resolve itself. > Pull in krb5_is_config_principal() when running against older kerberos > versions > --- > > Key: KUDU-2092 > URL: https://issues.apache.org/jira/browse/KUDU-2092 > Project: Kudu > Issue Type: Improvement > Components: security >Affects Versions: 1.4.0 >Reporter: Sailesh Mukil >Assignee: Sailesh Mukil > > On kerberos versions < krb5-1.8, the function krb5_is_config_principal() does > not exist. Since our code uses that function, and we dynamically link against > kerberos, we would be unable to build on systems that have these old kerberos > versions. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KUDU-2092) Pull in krb5_is_config_principal() when running against older kerberos versions
[ https://issues.apache.org/jira/browse/KUDU-2092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sailesh Mukil updated KUDU-2092: Description: On kerberos versions < krb5-1.8, the function krb5_is_config_principal() does not exist. Since our code uses that function, and we dynamically link against kerberos, we would be unable to build on systems that have these old kerberos versions. (was: On kerberos versions < krb5-1.8, the function krb5_is_config_principal() does not exist. Since our code uses that function, and we statically link against kerberos, we would be unable to run on systems that have these old kerberos versions.) > Pull in krb5_is_config_principal() when running against older kerberos > versions > --- > > Key: KUDU-2092 > URL: https://issues.apache.org/jira/browse/KUDU-2092 > Project: Kudu > Issue Type: Improvement > Components: security >Affects Versions: 1.4.0 >Reporter: Sailesh Mukil >Assignee: Sailesh Mukil > > On kerberos versions < krb5-1.8, the function krb5_is_config_principal() does > not exist. Since our code uses that function, and we dynamically link against > kerberos, we would be unable to build on systems that have these old kerberos > versions. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KUDU-2092) Pull in krb5_is_config_principal() when running against older kerberos versions
[ https://issues.apache.org/jira/browse/KUDU-2092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16119139#comment-16119139 ] Sailesh Mukil commented on KUDU-2092: - Oops, don't know why I wrote down statically, changed the description. Thanks for pointing them out. > Pull in krb5_is_config_principal() when running against older kerberos > versions > --- > > Key: KUDU-2092 > URL: https://issues.apache.org/jira/browse/KUDU-2092 > Project: Kudu > Issue Type: Improvement > Components: security >Affects Versions: 1.4.0 >Reporter: Sailesh Mukil >Assignee: Sailesh Mukil > > On kerberos versions < krb5-1.8, the function krb5_is_config_principal() does > not exist. Since our code uses that function, and we dynamically link against > kerberos, we would be unable to build on systems that have these old kerberos > versions. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (KUDU-2092) Pull in krb5_is_config_principal() when running against older kerberos versions
Sailesh Mukil created KUDU-2092: --- Summary: Pull in krb5_is_config_principal() when running against older kerberos versions Key: KUDU-2092 URL: https://issues.apache.org/jira/browse/KUDU-2092 Project: Kudu Issue Type: Improvement Components: security Affects Versions: 1.4.0 Reporter: Sailesh Mukil Assignee: Sailesh Mukil On kerberos versions < krb5-1.8, the function krb5_is_config_principal() does not exist. Since our code uses that function, and we statically link against kerberos, we would be unable to run on systems that have these old kerberos versions. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (KUDU-1929) [rpc] Allow using encrypted private keys for TLS
[ https://issues.apache.org/jira/browse/KUDU-1929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sailesh Mukil resolved KUDU-1929. - Resolution: Fixed Fix Version/s: 1.5.0 Commit in: https://github.com/apache/kudu/commit/57a07ae7217b63c51651611427f1af029d54d4fe > [rpc] Allow using encrypted private keys for TLS > > > Key: KUDU-1929 > URL: https://issues.apache.org/jira/browse/KUDU-1929 > Project: Kudu > Issue Type: Improvement > Components: rpc >Reporter: Sailesh Mukil >Assignee: Sailesh Mukil > Fix For: 1.5.0 > > > Currently, for internal RPC communication, we aren't able to handle encrypted > private keys. This can be done by using the OpenSSL APIs: > SSL_CTX_set_default_passwd_cb() > SSL_CTX_set_default_passwd_cb_userdata() -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (KUDU-1965) Allow user provided TLS certificates to work with KRPC
Sailesh Mukil created KUDU-1965: --- Summary: Allow user provided TLS certificates to work with KRPC Key: KUDU-1965 URL: https://issues.apache.org/jira/browse/KUDU-1965 Project: Kudu Issue Type: Bug Components: rpc, security Affects Versions: 1.3.0 Reporter: Sailesh Mukil Assignee: Sailesh Mukil Kudu generates certificates when TLS is enabled and if the user does not provide them. These certificates have Kudu specific encodings which are checked for in the RPC negotiation phase. If the certificates are user provided, it's highly likely that they will not contain these encodings, thus causing the certificate negotiation phase to fail. We can check for and verify these encodings if the certificates are provided by Kudu, else we can skip that step in the negotiation phase if the certificates are user provided. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (KUDU-1929) [rpc] Allow using encrypted private keys for TLS
Sailesh Mukil created KUDU-1929: --- Summary: [rpc] Allow using encrypted private keys for TLS Key: KUDU-1929 URL: https://issues.apache.org/jira/browse/KUDU-1929 Project: Kudu Issue Type: Improvement Components: rpc Reporter: Sailesh Mukil Assignee: Sailesh Mukil Currently, for internal RPC communication, we aren't able to handle encrypted private keys. This can be done by using the OpenSSL APIs: SSL_CTX_set_default_passwd_cb() SSL_CTX_set_default_passwd_cb_userdata() -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (KUDU-1865) Create fast path for RespondSuccess() in KRPC
[
https://issues.apache.org/jira/browse/KUDU-1865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15852466#comment-15852466
]
Sailesh Mukil commented on KUDU-1865:
-
CC [~henryr]
> Create fast path for RespondSuccess() in KRPC
> -
>
> Key: KUDU-1865
> URL: https://issues.apache.org/jira/browse/KUDU-1865
> Project: Kudu
> Issue Type: Improvement
> Components: rpc
>Reporter: Sailesh Mukil
> Labels: perfomance, rpc
>
> A lot of RPCs just respond with RespondSuccess() which returns the exact
> payload every time. This takes the same path as any other response by
> ultimately calling Connection::QueueResponseForCall() which has a few small
> allocations. These small allocations (and their corresponding deallocations)
> are called quite frequently (once for every IncomingCall) and end up taking
> quite some time in the kernel (traversing the free list, spin locks etc.)
> This was found when [~mmokhtar] ran some profiles on Impala over KRPC on a 20
> node cluster and found the following:
> The exact % of time spent is hard to quantify from the profiles, but these
> were the among the top 5 of the slowest stacks:
> {code:java}
> impalad ! tcmalloc::CentralFreeList::ReleaseToSpans - [unknown source file]
> impalad ! tcmalloc::CentralFreeList::ReleaseListToSpans + 0x1a - [unknown
> source file]
> impalad ! tcmalloc::CentralFreeList::InsertRange + 0x3b - [unknown source
> file]
> impalad ! tcmalloc::ThreadCache::ReleaseToCentralCache + 0x103 - [unknown
> source file]
> impalad ! tcmalloc::ThreadCache::Scavenge + 0x3e - [unknown source file]
> impalad ! operator delete + 0x329 - [unknown source file]
> impalad ! __gnu_cxx::new_allocator::deallocate + 0x4 -
> new_allocator.h:110
> impalad ! std::_Vector_base std::allocator>::_M_deallocate + 0x5 - stl_vector.h:178
> impalad ! ~_Vector_base + 0x4 - stl_vector.h:160
> impalad ! ~vector - stl_vector.h:425 'slices' vector
> impalad ! kudu::rpc::Connection::QueueResponseForCall + 0xac -
> connection.cc:433
> impalad ! kudu::rpc::InboundCall::Respond + 0xfa - inbound_call.cc:133
> impalad ! kudu::rpc::InboundCall::RespondSuccess + 0x43 - inbound_call.cc:77
> impalad ! kudu::rpc::RpcContext::RespondSuccess + 0x1f7 - rpc_context.cc:66
> ..
> {code}
> {code:java}
> impalad ! tcmalloc::CentralFreeList::FetchFromOneSpans - [unknown source file]
> impalad ! tcmalloc::CentralFreeList::RemoveRange + 0xc0 - [unknown source
> file]
> impalad ! tcmalloc::ThreadCache::FetchFromCentralCache + 0x62 - [unknown
> source file]
> impalad ! operator new + 0x297 - [unknown source file]<--- Creating
> new 'OutboundTransferTask' object.
> impalad ! kudu::rpc::Connection::QueueResponseForCall + 0x76 -
> connection.cc:432
> impalad ! kudu::rpc::InboundCall::Respond + 0xfa - inbound_call.cc:133
> impalad ! kudu::rpc::InboundCall::RespondSuccess + 0x43 - inbound_call.cc:77
> impalad ! kudu::rpc::RpcContext::RespondSuccess + 0x1f7 - rpc_context.cc:66
> ...
> {code}
> Even creating and deleting the 'RpcContext' takes a lot of time:
> {code:java}
> impalad ! tcmalloc::CentralFreeList::ReleaseToSpans - [unknown source file]
> impalad ! tcmalloc::CentralFreeList::ReleaseListToSpans + 0x1a - [unknown
> source file]
> impalad ! tcmalloc::CentralFreeList::InsertRange + 0x3b - [unknown source
> file]
> impalad ! tcmalloc::ThreadCache::ReleaseToCentralCache + 0x103 - [unknown
> source file]
> impalad ! tcmalloc::ThreadCache::Scavenge + 0x3e - [unknown source file]
> impalad ! operator delete + 0x329 - [unknown source file]
> impalad ! impala::TransmitDataResponsePb::~TransmitDataResponsePb + 0x16 -
> impala_internal_service.pb.cc:1221
> impalad ! impala::TransmitDataResponsePb::~TransmitDataResponsePb + 0x8 -
> impala_internal_service.pb.cc:1222
> impalad ! kudu::DefaultDeleter::operator() + 0x5 -
> gscoped_ptr.h:145
> impalad ! ~gscoped_ptr_impl + 0x9 - gscoped_ptr.h:228
> impalad ! ~gscoped_ptr - gscoped_ptr.h:318
> impalad ! kudu::rpc::RpcContext::~RpcContext + 0x1e - rpc_context.cc:53
> <-
> impalad ! kudu::rpc::RpcContext::RespondSuccess + 0x1ff - rpc_context.cc:67
> {code}
> The above show that creating these small objects under moderately heavy load
> results in heavy contention in the kernel. We will benefit a lot if we create
> a fast path for 'RespondSuccess'.
> My suggestion is to create all these small objects at once along with the
> 'InboundCall' object while it is being created, in a 'RespondSuccess'
> structure, and just use that structure when we want to send 'success' back to
> the sender. This would already contain the 'OutboundTransferTask', a 'Slice'
> with 'success', etc. We would expect that most RPCs respond with 'success' a
> majority of the time.
> How this would benefit us is that we
[jira] [Commented] (KUDU-1738) Allow users of C++ client to disable initialization of OpenSSL
[ https://issues.apache.org/jira/browse/KUDU-1738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15848859#comment-15848859 ] Sailesh Mukil commented on KUDU-1738: - [~tlipcon] Ah yes, my bad. That makes it more clear. I was just thinking about internal and external RPCs. This is a necessity even now then. > Allow users of C++ client to disable initialization of OpenSSL > -- > > Key: KUDU-1738 > URL: https://issues.apache.org/jira/browse/KUDU-1738 > Project: Kudu > Issue Type: Bug > Components: client, security >Affects Versions: 1.1.0 >Reporter: Todd Lipcon >Priority: Blocker > > As we start to use OpenSSL from within the client, we now take care of > initializing the OpenSSL library. Programs which embed us may have already > initialized OpenSSL for their own purposes. In that case, we don't want Kudu > to run the initialization a second time. > We should probably add some kind of option to KuduClientBuilder which > disables the SSL initialization. Another thought is that we may want to > detect if the SSL library has already been initialized and automatically > skip. (it seems this is possible by calling SSL_CTX_new and checking the > error code to see if it's not initialized yet) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (KUDU-1719) Heap use-after-free and deadlock on Messenger::Init() failure
[
https://issues.apache.org/jira/browse/KUDU-1719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sailesh Mukil resolved KUDU-1719.
-
Resolution: Fixed
Fix Version/s: 1.1.0
Commit in:
https://github.com/apache/kudu/commit/2426ef30650318bafd72cc49984aa7b5719b58e0
> Heap use-after-free and deadlock on Messenger::Init() failure
> -
>
> Key: KUDU-1719
> URL: https://issues.apache.org/jira/browse/KUDU-1719
> Project: Kudu
> Issue Type: Bug
>Affects Versions: 1.0.1
>Reporter: Sailesh Mukil
> Labels: easyfix
> Fix For: 1.1.0
>
>
> {code}
> Status MessengerBuilder::Build(shared_ptr *msgr) {
> ...
> gscoped_ptr new_msgr(new Messenger(*this));
> Status build_status = new_msgr->Init();
> if (!build_status.ok()) {
> new_msgr->AllExternalReferencesDropped();
> return build_status;
> }
> ...
> }
> {code}
> When new_msgr->Init() fails, AllExternalReferencesDropped() will be called
> which does a retain_self_.reset() which is a shared pointer to itself. If
> that was the last shared pointer reference to itself, it will call the
> destructor ~Messenger(). However, when the gscoped pointer 'new_msgr' goes
> out of scope, it too will call the destructor, but by this point, all the
> members have already been freed causing a use-after-free.
>
> When Messenger::Init() fails before all the reactors have been Init()'ed, the
> reactors that haven't been Init()'ed do not give up their reference to the
> Messenger object. This is because this reference is only given up in
> ReactorThread::RunThread(), which is not called unless reactor->Init() is
> called:
> {code}
> void ReactorThread::RunThread() {
> ...
> DVLOG(6) << "Calling ReactorThread::RunThread()...";
> loop_.run(0);
> VLOG(1) << name() << " thread exiting.";
> // No longer need the messenger. This causes the messenger to
> // get deleted when all the reactors exit.
> reactor_->messenger_.reset();
> }
> {code}
> So when the gscoped_ptr 'new_msgr' goes out of scope, it will call the
> destructor which calls STLDeleteElements(_); where the Messenger
> references are finally dropped. But when the last Messenger reference is
> dropped, that too will call the destructor ~Messenger() which gets stuck on
> the lock:
> {code}
> Messenger::~Messenger() {
> std::lock_guard guard(lock_); // Gets stuck on this lock
> CHECK(closing_) << "Should have already shut down";
> STLDeleteElements(_); // This will call the ~Messenger()
> destructor again
> }
> {code}
> The easy fix for this is to make the gscoped_ptr 'new_msgr' a raw pointer
> instead, and not explicitly free it since it will be freed by the shared
> pointers pointing to it. (But this may look like buggy code, not sure if
> there's a better way)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
