[
https://issues.apache.org/jira/browse/KNOX-1221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16645482#comment-16645482
]
Kevin Risden commented on KNOX-1221:
------------------------------------
I was able to reproduce this. There are a few factors in play:
* Read
** Native
** Non TLS/SSL Knox
** TLS/SSL Knox
* Write
** Native
** Non TLS/SSL Knox
** TLS/SSL Knox
The following results are tested with:
* a single 4 core 4GB RAM Centos 7 VM on my Macbook Pro laptop
* openjdk version "1.8.0_181"
* Hadoop 3.1.1 single node pseudo distributed
* Knox 1.1.0
* 1GB random file
* Read/write over WebHDFS
Testing commands
* Random file creation
** dd if=/dev/zero of=file.txt count=1024 bs=1048576
* Read
** Native
*** wget "http://node.vagrant:9870/webhdfs/v1/file.txt?op=OPEN"; -O
/opt/webhdfsfile.txt
** Non TLS/SSL Knox
*** wget --user=admin --password=admin-password --no-check-certificate
"http://node.vagrant:8443/gateway/sandbox/webhdfs/v1/file.txt?op=OPEN"; -O
/opt/knoxfile.txt
** TLS/SSL Knox
*** wget --user=admin --password=admin-password --no-check-certificate
"https://node.vagrant:8443/gateway/sandbox/webhdfs/v1/file.txt?op=OPEN"; -O
/opt/knoxfile.txt
* Write
** Native
*** time curl --progress-bar --verbose -k -u admin:admin-password
--location-trusted -i -XPUT -T /opt/randomfile/file.txt
"http://node.vagrant:9870/webhdfs/v1/admin/randomfileupload.txt?op=CREATE&overwrite=true&user.name=admin";
** Non TLS/SSL Knox
*** time curl --progress-bar --verbose -k -u admin:admin-password
--location-trusted -i -XPUT -T /opt/randomfile/file.txt
"http://node.vagrant:8443/gateway/sandbox/webhdfs/v1/admin/randomfileupload.txt?op=CREATE&overwrite=true";
** TLS/SSL Knox
*** time curl --progress-bar --verbose -k -u admin:admin-password
--location-trusted -i -XPUT -T /opt/randomfile/file.txt
"https://node.vagrant:8443/gateway/sandbox/webhdfs/v1/admin/randomfileupload.txt?op=CREATE&overwrite=true";
Read performance
* Single read
** Native WebHDFS - 252MB/s in 3.8s
** Non TLS/SSL Knox - 264MB/s in 3.6s
** TLS/SSL Knox - 54.3MB/s in 20s
* Parallel read
** TLS/SSL Knox
***
1,073,741,824 46.1MB/s in 22s
***
1,073,741,824 51.3MB/s in 22s
Write performance
* Native WebHDFS - ~2.6 seconds
* Non TLS/SSL Knox - ~29 seconds
* TLS/SSL Knox - ~49.6 seconds
> WebHDFS read/write performance limitations
> ------------------------------------------
>
> Key: KNOX-1221
> URL: https://issues.apache.org/jira/browse/KNOX-1221
> Project: Apache Knox
> Issue Type: Bug
> Components: Server
> Affects Versions: 0.9.0, 0.10.0, 0.11.0, 0.12.0, 0.13.0, 0.14.0, 1.0.0,
> 1.1.0
> Environment: Knox 1.0.0
> RHEL 6.9
> JDK 1.8.0_60
> Launch string:
> /usr/bin/java -Djava.library.path=/opt/knox-1.0.0/ext/native -Xms2g -Xmx8g
> -Dcom.sun.management.jmxremote
> -Dcom.sun.management.jmxremote.authenticate=false
> -Dcom.sun.management.jmxremote.ssl=false
> -Dcom.sun.management.jmxremote.port=9000 -jar /opt/knox-1.0.0/bin/gateway.jar
> Reporter: Georgy
> Assignee: Kevin Risden
> Priority: Major
> Fix For: 1.2.0
>
> Attachments: gateway-site.xml, rtk.xml
>
>
> If I use direct connection to WebHDFS from one node I have speed nearly
> several gigabites/sec when download or upload large files. But if I use knox
> I have ulpload/download speed only 100mbit/sec from the same node. Found that
> knox limits speed for one https session. As a workaround I upload large files
> through knox in parallel but it is not a good way.
> Couldn't find any configuration params that can control such behaviour. Is it
> a code limitation?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
