[
https://issues.apache.org/jira/browse/HADOOP-14558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mostafa Mokhtar updated HADOOP-14558:
-------------------------------------
Attachment: hdfs_secure_namenode_16_threads_194363.jfr
> RPC requests on a secure cluster are 10x slower due to expensive encryption
> and decryption
> -------------------------------------------------------------------------------------------
>
> Key: HADOOP-14558
> URL: https://issues.apache.org/jira/browse/HADOOP-14558
> Project: Hadoop Common
> Issue Type: Bug
> Components: hdfs-client
> Affects Versions: 2.6.0
> Reporter: Mostafa Mokhtar
> Priority: Critical
> Labels: impala, metadata, rpc
> Attachments: hdfs_secure_namenode_16_threads_194363.jfr
>
>
> While running a performance tests for Impala comparing secure and un-secure
> clusters I noticed that metadata loading operations are 10x slower on a
> cluster with Kerberos+SSL enabled.
> hadoop.rpc.protection is set to privacy
> Any recommendations on how this can be mitigated? 10x slowdown is a big hit
> for metadata loading.
> The majority of the slowdown is coming from the two threads below.
> {code}
> Stack Trace Sample Count Percentage(%)
> org.apache.hadoop.ipc.Client$Connection.run() 5,212 46.586
> org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse() 5,203
> 46.505
> java.io.DataInputStream.readInt() 5,039 45.039
> java.io.BufferedInputStream.read() 5,038 45.03
> java.io.BufferedInputStream.fill() 5,038 45.03
>
> org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(byte[], int,
> int) 5,036 45.013
> java.io.FilterInputStream.read(byte[], int, int) 5,036
> 45.013
>
> org.apache.hadoop.security.SaslRpcClient$WrappedInputStream.read(byte[], int,
> int) 5,036 45.013
>
> org.apache.hadoop.security.SaslRpcClient$WrappedInputStream.readNextRpcPacket()
> 5,035 45.004
>
> com.sun.security.sasl.gsskerb.GssKrb5Base.unwrap(byte[], int, int) 4,775
> 42.68
> sun.security.jgss.GSSContextImpl.unwrap(byte[],
> int, int, MessageProp) 4,775 42.68
>
> sun.security.jgss.krb5.Krb5Context.unwrap(byte[], int, int, MessageProp)
> 4,768 42.617
>
> sun.security.jgss.krb5.WrapToken.getData() 4,714 42.134
>
> sun.security.jgss.krb5.WrapToken.getData(byte[], int) 4,714 42.134
>
> sun.security.jgss.krb5.WrapToken.getDataFromBuffer(byte[], int) 4,714
> 42.134
>
> sun.security.jgss.krb5.CipherHelper.decryptData(WrapToken, byte[], int, int,
> byte[], int) 3,083 27.556
>
> sun.security.jgss.krb5.CipherHelper.des3KdDecrypt(WrapToken, byte[], int,
> int, byte[], int) 3,078 27.512
>
> sun.security.krb5.internal.crypto.Des3.decryptRaw(byte[], int, byte[],
> byte[], int, int) 3,076 27.494
>
> sun.security.krb5.internal.crypto.dk.DkCrypto.decryptRaw(byte[], int, byte[],
> byte[], int, int) 3,076 27.494
> {code}
> And
> {code}
> Stack Trace Sample Count Percentage(%)
> java.lang.Thread.run() 3,379 30.202
> java.util.concurrent.ThreadPoolExecutor$Worker.run() 3,379 30.202
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker)
> 3,379 30.202
> java.util.concurrent.FutureTask.run() 3,367 30.095
> java.util.concurrent.Executors$RunnableAdapter.call() 3,367
> 30.095
> org.apache.hadoop.ipc.Client$Connection$3.run() 3,367
> 30.095
> java.io.DataOutputStream.flush() 3,367 30.095
> java.io.BufferedOutputStream.flush() 3,367 30.095
> java.io.BufferedOutputStream.flushBuffer() 3,367
> 30.095
>
> org.apache.hadoop.security.SaslRpcClient$WrappedOutputStream.write(byte[],
> int, int) 3,367 30.095
>
> com.sun.security.sasl.gsskerb.GssKrb5Base.wrap(byte[], int, int) 3,281
> 29.326
>
> sun.security.jgss.GSSContextImpl.wrap(byte[], int, int, MessageProp) 3,281
> 29.326
>
> sun.security.jgss.krb5.Krb5Context.wrap(byte[], int, int, MessageProp)
> 3,280 29.317
>
> sun.security.jgss.krb5.WrapToken.<init>(Krb5Context, MessageProp, byte[],
> int, int) 1,872 16.732
>
> sun.security.jgss.krb5.WrapToken.encode() 1,407 12.576
> {code}
> This is the Impala Catalog thread which initiates the NameNode request
> {code}
> Stack Trace Sample Count Percentage(%)
> org.apache.impala.service.JniCatalog.resetMetadata(byte[]) 2,414 21.577
>
> org.apache.impala.service.CatalogOpExecutor.execResetMetadata(TResetMetadataRequest)
> 2,378 21.255
> org.apache.impala.catalog.CatalogServiceCatalog.reloadTable(Table)
> 2,378 21.255
> org.apache.impala.catalog.HdfsTable.load(boolean, IMetaStoreClient,
> Table) 2,351 21.014
> org.apache.impala.catalog.HdfsTable.load(boolean,
> IMetaStoreClient, Table, boolean, boolean, Set) 2,351 21.014
>
> org.apache.impala.catalog.HdfsTable.updatePartitionsFromHms(IMetaStoreClient,
> Set, boolean) 2,350 21.005
>
> org.apache.impala.catalog.HdfsTable.loadPartitionFileMetadata(List) 2,326
> 20.79
>
> org.apache.impala.catalog.HdfsTable.loadPartitionFileMetadata(StorageDescriptor,
> HdfsPartition) 2,233 19.959
>
> org.apache.impala.catalog.HdfsTable.refreshFileMetadata(HdfsPartition)
> 1,998 17.858
>
> org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(Path) 1,496
> 13.371
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
