[
https://issues.apache.org/jira/browse/KNOX-1221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16645490#comment-16645490
]
Kevin Risden edited comment on KNOX-1221 at 10/10/18 8:25 PM:
--------------------------------------------------------------
h2. Write Performance Difference Explanation
One of the troubling outcomes of the testing was that for writes there was a
significant difference between native WebHDFS and even non TLS/SSL Knox. The
results are highlighted again below
* Native WebHDFS - ~2.6 seconds
* Non TLS/SSL Knox - ~29 seconds
* TLS/SSL Knox - ~49.6 seconds
The TLS/SSL vs Non TLS/SSL difference of ~20 seconds can be attributed to the
description above. There is no explanation in the JDK why there is a ~26 second
difference between native and Non TLS/SSL Knox.
I investigated further and found that there is a Knox bottleneck on the write
side performance of WebHDFS. Knox isn't streaming the file quickly and instead
is doing a single byte read at a time.
[UrlRewriteRequestStream|https://github.com/apache/knox/blob/bcb2e1f0196548d9cc54ca94e8fe79198ef61906/gateway-provider-rewrite/src/main/java/org/apache/knox/gateway/filter/rewrite/impl/UrlRewriteRequestStream.java]
doesn't implement the more efficient read methods for the InputStream. This
means that there is a significant performance hit when writing data through
Knox. I tested locally with the two additional read methods implemented and the
results were:
* Native WebHDFS - ~3.3 seconds
* Non TLS/SSL Knox - ~4.2 seconds
* TLS/SSL Knox - ~4.8 seconds (Note this had the Conscrypt implementation as
well)
This is a drastic ~85% reduction in time for writing a 1GB file. I created
KNOX-1521 to implement this change to improve streaming performance.
Some things that were tried that didn't help write performance:
* Increasing buffer sizes
** httpcore ChunkedOuputStream increase from hardcoded 2048
** httpcore fragmentation size hint increase from 8192
** httpcore connection config buffer increase from 8192
** Knox InputEntityStream buffer size increase from 16384
** Upgrade to httpclient 5/httpcore 5
was (Author: risdenk):
h2. Write Performance Difference Explanation
One of the troubling outcomes of the testing was that for writes there was a
significant difference between native WebHDFS and even non TLS/SSL Knox. The
results are highlighted again below
* Native WebHDFS - ~2.6 seconds
* Non TLS/SSL Knox - ~29 seconds
* TLS/SSL Knox - ~49.6 seconds
The TLS/SSL vs Non TLS/SSL difference of ~20 seconds can be attributed to the
description above. There is no explanation in the JDK why there is a ~26 second
difference between native and Non TLS/SSL Knox.
I investigated further and found that there is a Knox bottleneck on the write
side performance of WebHDFS. Knox isn't streaming the file quickly and instead
is doing a single byte read at a time.
[UrlRewriteRequestStream|https://github.com/apache/knox/blob/bcb2e1f0196548d9cc54ca94e8fe79198ef61906/gateway-provider-rewrite/src/main/java/org/apache/knox/gateway/filter/rewrite/impl/UrlRewriteRequestStream.java]
doesn't implement the more efficient read methods for the InputStream. This
means that there is a significant performance hit when writing data through
Knox. I tested locally with the two additional read methods implemented and the
results were:
* Native WebHDFS - ~3.3 seconds
* Non TLS/SSL Knox - ~4.2 seconds
* TLS/SSL Knox - ~4.8 seconds (Note this had the Conscrypt implementation as
well)
This is a drastic ~85% reduction in time for writing a 1GB file. I created
KNOX-1521 to implement this change to improve streaming performance.
> WebHDFS read/write performance limitations
> ------------------------------------------
>
> Key: KNOX-1221
> URL: https://issues.apache.org/jira/browse/KNOX-1221
> Project: Apache Knox
> Issue Type: Bug
> Components: Server
> Affects Versions: 0.9.0, 0.10.0, 0.11.0, 0.12.0, 0.13.0, 0.14.0, 1.0.0,
> 1.1.0
> Environment: Knox 1.0.0
> RHEL 6.9
> JDK 1.8.0_60
> Launch string:
> /usr/bin/java -Djava.library.path=/opt/knox-1.0.0/ext/native -Xms2g -Xmx8g
> -Dcom.sun.management.jmxremote
> -Dcom.sun.management.jmxremote.authenticate=false
> -Dcom.sun.management.jmxremote.ssl=false
> -Dcom.sun.management.jmxremote.port=9000 -jar /opt/knox-1.0.0/bin/gateway.jar
> Reporter: Georgy
> Assignee: Kevin Risden
> Priority: Major
> Fix For: 1.2.0
>
> Attachments: gateway-site.xml, rtk.xml
>
>
> If I use direct connection to WebHDFS from one node I have speed nearly
> several gigabites/sec when download or upload large files. But if I use knox
> I have ulpload/download speed only 100mbit/sec from the same node. Found that
> knox limits speed for one https session. As a workaround I upload large files
> through knox in parallel but it is not a good way.
> Couldn't find any configuration params that can control such behaviour. Is it
> a code limitation?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
