[
https://issues.apache.org/jira/browse/HADOOP-14558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16057343#comment-16057343
]
Steve Loughran edited comment on HADOOP-14558 at 6/21/17 11:28 AM:
-------------------------------------------------------------------
Duplicate of HADOOP-10768.
Problem here is that the java encryption APIs allocate new byte buffers, get
expensive fast.
If it were to be fixed, the obvious solution would be to have an option for
native encryption and a private API which would encrypt either in-place or to a
preallocated buffer, the latter being able to compensate for the fact that
encrypted blocks often get their sizes rounded up.
looks the like HADOOP-10768 patch could be viable, so its a matter of
marshalling all the IPC reviewers needed to be happy with it. It's a big patch
to a core piece of the code, so will no doubt face inertia —but would be
invaluable
was (Author: [email protected]):
Duplicate of HADOOP-10768.
Problem here is that the java encryption APIs allocate new byte buffers, get
expensive fast.
If it were to be fixed, the obvious solution would be to have an option for
native encryption and a private API which would encrypt either in-place or to a
preallocated buffer, the latter being able to compensate for the fact that
encrypted blocks often get their sizes rounded up. Talk to the intel developers
and see if they have time to play with this?
> RPC requests on a secure cluster are 10x slower due to expensive encryption
> and decryption
> -------------------------------------------------------------------------------------------
>
> Key: HADOOP-14558
> URL: https://issues.apache.org/jira/browse/HADOOP-14558
> Project: Hadoop Common
> Issue Type: Bug
> Components: hdfs-client
> Affects Versions: 2.6.0
> Reporter: Mostafa Mokhtar
> Priority: Critical
> Labels: impala, metadata, rpc
>
> While running a performance tests for Impala comparing secure and un-secure
> clusters I noticed that metadata loading operations are 10x slower on a
> cluster with Kerberos+SSL enabled.
> hadoop.rpc.protection is set to privacy
> Any recommendations on how this can be mitigated? 10x slowdown is a big hit
> for metadata loading.
> The majority of the slowdown is coming from the two threads below.
> {code}
> Stack Trace Sample Count Percentage(%)
> org.apache.hadoop.ipc.Client$Connection.run() 5,212 46.586
> org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse() 5,203
> 46.505
> java.io.DataInputStream.readInt() 5,039 45.039
> java.io.BufferedInputStream.read() 5,038 45.03
> java.io.BufferedInputStream.fill() 5,038 45.03
>
> org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(byte[], int,
> int) 5,036 45.013
> java.io.FilterInputStream.read(byte[], int, int) 5,036
> 45.013
>
> org.apache.hadoop.security.SaslRpcClient$WrappedInputStream.read(byte[], int,
> int) 5,036 45.013
>
> org.apache.hadoop.security.SaslRpcClient$WrappedInputStream.readNextRpcPacket()
> 5,035 45.004
>
> com.sun.security.sasl.gsskerb.GssKrb5Base.unwrap(byte[], int, int) 4,775
> 42.68
> sun.security.jgss.GSSContextImpl.unwrap(byte[],
> int, int, MessageProp) 4,775 42.68
>
> sun.security.jgss.krb5.Krb5Context.unwrap(byte[], int, int, MessageProp)
> 4,768 42.617
>
> sun.security.jgss.krb5.WrapToken.getData() 4,714 42.134
>
> sun.security.jgss.krb5.WrapToken.getData(byte[], int) 4,714 42.134
>
> sun.security.jgss.krb5.WrapToken.getDataFromBuffer(byte[], int) 4,714
> 42.134
>
> sun.security.jgss.krb5.CipherHelper.decryptData(WrapToken, byte[], int, int,
> byte[], int) 3,083 27.556
>
> sun.security.jgss.krb5.CipherHelper.des3KdDecrypt(WrapToken, byte[], int,
> int, byte[], int) 3,078 27.512
>
> sun.security.krb5.internal.crypto.Des3.decryptRaw(byte[], int, byte[],
> byte[], int, int) 3,076 27.494
>
> sun.security.krb5.internal.crypto.dk.DkCrypto.decryptRaw(byte[], int, byte[],
> byte[], int, int) 3,076 27.494
> {code}
> And
> {code}
> Stack Trace Sample Count Percentage(%)
> java.lang.Thread.run() 3,379 30.202
> java.util.concurrent.ThreadPoolExecutor$Worker.run() 3,379 30.202
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker)
> 3,379 30.202
> java.util.concurrent.FutureTask.run() 3,367 30.095
> java.util.concurrent.Executors$RunnableAdapter.call() 3,367
> 30.095
> org.apache.hadoop.ipc.Client$Connection$3.run() 3,367
> 30.095
> java.io.DataOutputStream.flush() 3,367 30.095
> java.io.BufferedOutputStream.flush() 3,367 30.095
> java.io.BufferedOutputStream.flushBuffer() 3,367
> 30.095
>
> org.apache.hadoop.security.SaslRpcClient$WrappedOutputStream.write(byte[],
> int, int) 3,367 30.095
>
> com.sun.security.sasl.gsskerb.GssKrb5Base.wrap(byte[], int, int) 3,281
> 29.326
>
> sun.security.jgss.GSSContextImpl.wrap(byte[], int, int, MessageProp) 3,281
> 29.326
>
> sun.security.jgss.krb5.Krb5Context.wrap(byte[], int, int, MessageProp)
> 3,280 29.317
>
> sun.security.jgss.krb5.WrapToken.<init>(Krb5Context, MessageProp, byte[],
> int, int) 1,872 16.732
>
> sun.security.jgss.krb5.WrapToken.encode() 1,407 12.576
> {code}
> This is the Impala Catalog thread which initiates the NameNode request
> {code}
> Stack Trace Sample Count Percentage(%)
> org.apache.impala.service.JniCatalog.resetMetadata(byte[]) 2,414 21.577
>
> org.apache.impala.service.CatalogOpExecutor.execResetMetadata(TResetMetadataRequest)
> 2,378 21.255
> org.apache.impala.catalog.CatalogServiceCatalog.reloadTable(Table)
> 2,378 21.255
> org.apache.impala.catalog.HdfsTable.load(boolean, IMetaStoreClient,
> Table) 2,351 21.014
> org.apache.impala.catalog.HdfsTable.load(boolean,
> IMetaStoreClient, Table, boolean, boolean, Set) 2,351 21.014
>
> org.apache.impala.catalog.HdfsTable.updatePartitionsFromHms(IMetaStoreClient,
> Set, boolean) 2,350 21.005
>
> org.apache.impala.catalog.HdfsTable.loadPartitionFileMetadata(List) 2,326
> 20.79
>
> org.apache.impala.catalog.HdfsTable.loadPartitionFileMetadata(StorageDescriptor,
> HdfsPartition) 2,233 19.959
>
> org.apache.impala.catalog.HdfsTable.refreshFileMetadata(HdfsPartition)
> 1,998 17.858
>
> org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(Path) 1,496
> 13.371
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
