[
https://issues.apache.org/jira/browse/KNOX-2139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17046940#comment-17046940
]
Kevin Risden commented on KNOX-2139:
------------------------------------
I also wasn't able to reproduce with 8GB file:
{code:java}
[root@krisden-1 ~]# dd if=/dev/zero of=data8gb.bin bs=1024 count=8388608
8388608+0 records in
8388608+0 records out
8589934592 bytes (8.6 GB) copied, 25.9844 s, 331 MB/s
[root@krisden-1 ~]# ls -l data8gb.bin
-rw-r--r-- 1 root root 8589934592 Feb 27 19:45 data8gb.bin
[root@krisden-1 ~]# time curl -v -L --location-trusted -iku systest:cloudera -X
PUT -T data8gb.bin
"https://krisden-1.krisden.root.hwx.site:8443/gateway/cdp-proxy-api/webhdfs/v1/tmp/data8gb.bin?op=CREATE&overwrite=true";
* About to connect() to krisden-1.krisden.root.hwx.site port 8443 (#0)
* Trying 172.27.157.134...
* Connected to krisden-1.krisden.root.hwx.site (172.27.157.134) port 8443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* skipping SSL peer certificate verification
* SSL connection using TLS_DHE_RSA_WITH_AES_256_CBC_SHA256
* Server certificate:
* subject: CN=localhost,OU=Test,O=Hadoop,L=Test,ST=Test,C=US
* start date: Feb 27 19:08:52 2020 GMT
* expire date: Feb 26 19:08:52 2021 GMT
* common name: localhost
* issuer: CN=localhost,OU=Test,O=Hadoop,L=Test,ST=Test,C=US
* Server auth using Basic with user 'systest'
> PUT
> /gateway/cdp-proxy-api/webhdfs/v1/tmp/data8gb.bin?op=CREATE&overwrite=true
> HTTP/1.1
> Authorization: Basic c3lzdGVzdDpjbG91ZGVyYQ==
> User-Agent: curl/7.29.0
> Host: krisden-1.krisden.root.hwx.site:8443
> Accept: */*
> Content-Length: 8589934592
> Expect: 100-continue
>
< HTTP/1.1 307 Temporary Redirect
HTTP/1.1 307 Temporary Redirect
< Date: Thu, 27 Feb 2020 19:46:33 GMT
Date: Thu, 27 Feb 2020 19:46:33 GMT
< Set-Cookie: KNOXSESSIONID=node010t0ve1yaixh4qnh7gz8wweew10.node0;
Path=/gateway/cdp-proxy-api; Secure; HttpOnly
Set-Cookie: KNOXSESSIONID=node010t0ve1yaixh4qnh7gz8wweew10.node0;
Path=/gateway/cdp-proxy-api; Secure; HttpOnly
< Expires: Thu, 01 Jan 1970 00:00:00 GMT
Expires: Thu, 01 Jan 1970 00:00:00 GMT
< Set-Cookie: rememberMe=deleteMe; Path=/gateway/cdp-proxy-api; Max-Age=0;
Expires=Wed, 26-Feb-2020 19:46:34 GMT
Set-Cookie: rememberMe=deleteMe; Path=/gateway/cdp-proxy-api; Max-Age=0;
Expires=Wed, 26-Feb-2020 19:46:34 GMT
< Date: Thu, 27 Feb 2020 19:46:34 GMT
Date: Thu, 27 Feb 2020 19:46:34 GMT
< Cache-Control: no-cache
Cache-Control: no-cache
< Expires: Thu, 27 Feb 2020 19:46:34 GMT
Expires: Thu, 27 Feb 2020 19:46:34 GMT
< Date: Thu, 27 Feb 2020 19:46:34 GMT
Date: Thu, 27 Feb 2020 19:46:34 GMT
< Pragma: no-cache
Pragma: no-cache
< X-Content-Type-Options: nosniff
X-Content-Type-Options: nosniff
< X-FRAME-OPTIONS: SAMEORIGIN
X-FRAME-OPTIONS: SAMEORIGIN
< X-XSS-Protection: 1; mode=block
X-XSS-Protection: 1; mode=block
< Location:
https://krisden-1.krisden.root.hwx.site:8443/gateway/cdp-proxy-api/webhdfs/data/v1/webhdfs/v1/tmp/data8gb.bin?_=AAAACAAAABAAAAFAxKrs3nk3X0u7q-dLDKW0de2fMdBbGow4dOTkq1ZOawxV0OrmrYdu0n9TbnouLaq6CjsMOGARlhElWJvilqVPf3_LjXqVSRFMjfW-conltMCpBLWyBKDdSENI2sGBSd4FtlBeWZWZoCkkOwhnXeeQ8XgQH_sE8-zZfFvZOmdjTA0GfLb3esh6tYdUv8bZLRYDw6v8VY_tHkTEukecSlUM5r2mdMryMfJk70cxVy5t8Zhenat44RZlz9QEZezFiv1xlc3kvLhhlDuYLnDHHwL-Z3Cd6rECfTeOXKTJlRCaPHkQWUfDg65Bp9QZ-ZoGsFUQ3pbmSqmDXVKVyJLP41I7AGvjN_p3fxPP1QBTTQXWMZ3ZsepKtRAsaGlc3E8qgXMkJRrOzifYV4ERDGSSrveNeaRu1PPx8VQcPOGOL5cNd-41_NMiBv23GA6Wf-TarDuGwz5EA5hSajo
Location:
https://krisden-1.krisden.root.hwx.site:8443/gateway/cdp-proxy-api/webhdfs/data/v1/webhdfs/v1/tmp/data8gb.bin?_=AAAACAAAABAAAAFAxKrs3nk3X0u7q-dLDKW0de2fMdBbGow4dOTkq1ZOawxV0OrmrYdu0n9TbnouLaq6CjsMOGARlhElWJvilqVPf3_LjXqVSRFMjfW-conltMCpBLWyBKDdSENI2sGBSd4FtlBeWZWZoCkkOwhnXeeQ8XgQH_sE8-zZfFvZOmdjTA0GfLb3esh6tYdUv8bZLRYDw6v8VY_tHkTEukecSlUM5r2mdMryMfJk70cxVy5t8Zhenat44RZlz9QEZezFiv1xlc3kvLhhlDuYLnDHHwL-Z3Cd6rECfTeOXKTJlRCaPHkQWUfDg65Bp9QZ-ZoGsFUQ3pbmSqmDXVKVyJLP41I7AGvjN_p3fxPP1QBTTQXWMZ3ZsepKtRAsaGlc3E8qgXMkJRrOzifYV4ERDGSSrveNeaRu1PPx8VQcPOGOL5cNd-41_NMiBv23GA6Wf-TarDuGwz5EA5hSajo
< Content-Type: application/octet-stream
Content-Type: application/octet-stream
< Content-Length: 0
Content-Length: 0
< Connection: close
Connection: close
< Server: Jetty(9.4.26.v20200117)
Server: Jetty(9.4.26.v20200117)
<
* Closing connection 0
* Issue another request to this URL:
'https://krisden-1.krisden.root.hwx.site:8443/gateway/cdp-proxy-api/webhdfs/data/v1/webhdfs/v1/tmp/data8gb.bin?_=AAAACAAAABAAAAFAxKrs3nk3X0u7q-dLDKW0de2fMdBbGow4dOTkq1ZOawxV0OrmrYdu0n9TbnouLaq6CjsMOGARlhElWJvilqVPf3_LjXqVSRFMjfW-conltMCpBLWyBKDdSENI2sGBSd4FtlBeWZWZoCkkOwhnXeeQ8XgQH_sE8-zZfFvZOmdjTA0GfLb3esh6tYdUv8bZLRYDw6v8VY_tHkTEukecSlUM5r2mdMryMfJk70cxVy5t8Zhenat44RZlz9QEZezFiv1xlc3kvLhhlDuYLnDHHwL-Z3Cd6rECfTeOXKTJlRCaPHkQWUfDg65Bp9QZ-ZoGsFUQ3pbmSqmDXVKVyJLP41I7AGvjN_p3fxPP1QBTTQXWMZ3ZsepKtRAsaGlc3E8qgXMkJRrOzifYV4ERDGSSrveNeaRu1PPx8VQcPOGOL5cNd-41_NMiBv23GA6Wf-TarDuGwz5EA5hSajo'
* About to connect() to krisden-1.krisden.root.hwx.site port 8443 (#1)
* Trying 172.27.157.134...
* Connected to krisden-1.krisden.root.hwx.site (172.27.157.134) port 8443 (#1)
* skipping SSL peer certificate verification
* SSL connection using TLS_DHE_RSA_WITH_AES_256_CBC_SHA256
* Server certificate:
* subject: CN=localhost,OU=Test,O=Hadoop,L=Test,ST=Test,C=US
* start date: Feb 27 19:08:52 2020 GMT
* expire date: Feb 26 19:08:52 2021 GMT
* common name: localhost
* issuer: CN=localhost,OU=Test,O=Hadoop,L=Test,ST=Test,C=US
* Server auth using Basic with user 'systest'
> PUT
> /gateway/cdp-proxy-api/webhdfs/data/v1/webhdfs/v1/tmp/data8gb.bin?_=AAAACAAAABAAAAFAxKrs3nk3X0u7q-dLDKW0de2fMdBbGow4dOTkq1ZOawxV0OrmrYdu0n9TbnouLaq6CjsMOGARlhElWJvilqVPf3_LjXqVSRFMjfW-conltMCpBLWyBKDdSENI2sGBSd4FtlBeWZWZoCkkOwhnXeeQ8XgQH_sE8-zZfFvZOmdjTA0GfLb3esh6tYdUv8bZLRYDw6v8VY_tHkTEukecSlUM5r2mdMryMfJk70cxVy5t8Zhenat44RZlz9QEZezFiv1xlc3kvLhhlDuYLnDHHwL-Z3Cd6rECfTeOXKTJlRCaPHkQWUfDg65Bp9QZ-ZoGsFUQ3pbmSqmDXVKVyJLP41I7AGvjN_p3fxPP1QBTTQXWMZ3ZsepKtRAsaGlc3E8qgXMkJRrOzifYV4ERDGSSrveNeaRu1PPx8VQcPOGOL5cNd-41_NMiBv23GA6Wf-TarDuGwz5EA5hSajo
> HTTP/1.1
> Authorization: Basic c3lzdGVzdDpjbG91ZGVyYQ==
> User-Agent: curl/7.29.0
> Host: krisden-1.krisden.root.hwx.site:8443
> Accept: */*
> Content-Length: 8589934592
> Expect: 100-continue
>
< HTTP/1.1 100 Continue
HTTP/1.1 100 Continue
* We are completely uploaded and fine
< HTTP/1.1 201 Created
HTTP/1.1 201 Created
< Date: Thu, 27 Feb 2020 19:46:34 GMT
Date: Thu, 27 Feb 2020 19:46:34 GMT
< Set-Cookie: KNOXSESSIONID=node01jnbdu8qvxjshqfave431j4rp11.node0;
Path=/gateway/cdp-proxy-api; Secure; HttpOnly
Set-Cookie: KNOXSESSIONID=node01jnbdu8qvxjshqfave431j4rp11.node0;
Path=/gateway/cdp-proxy-api; Secure; HttpOnly
< Expires: Thu, 01 Jan 1970 00:00:00 GMT
Expires: Thu, 01 Jan 1970 00:00:00 GMT
< Set-Cookie: rememberMe=deleteMe; Path=/gateway/cdp-proxy-api; Max-Age=0;
Expires=Wed, 26-Feb-2020 19:46:36 GMT
Set-Cookie: rememberMe=deleteMe; Path=/gateway/cdp-proxy-api; Max-Age=0;
Expires=Wed, 26-Feb-2020 19:46:36 GMT
< Location:
https://krisden-1.krisden.root.hwx.site:8443/gateway/cdp-proxy-api/webhdfs/v1/tmp/data8gb.bin
Location:
https://krisden-1.krisden.root.hwx.site:8443/gateway/cdp-proxy-api/webhdfs/v1/tmp/data8gb.bin
< Access-Control-Allow-Origin: *
Access-Control-Allow-Origin: *
< Connection: close
Connection: close
< Server: Jetty(9.4.26.v20200117)
Server: Jetty(9.4.26.v20200117)
<
* Closing connection 1
real 1m48.136s
user 1m35.650s
sys 0m6.469s
[root@krisden-1 ~]# curl -iku systest:cloudera
"https://krisden-1.krisden.root.hwx.site:8443/gateway/cdp-proxy-api/webhdfs/v1/tmp/data8gb.bin?op=LISTSTATUS";
HTTP/1.1 200 OK
Date: Thu, 27 Feb 2020 19:48:29 GMT
Set-Cookie: KNOXSESSIONID=node0cpzqyshxfa211eimbrqamlovt12.node0;
Path=/gateway/cdp-proxy-api; Secure; HttpOnly
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Set-Cookie: rememberMe=deleteMe; Path=/gateway/cdp-proxy-api; Max-Age=0;
Expires=Wed, 26-Feb-2020 19:48:31 GMT
Date: Thu, 27 Feb 2020 19:48:31 GMT
Cache-Control: no-cache
Expires: Thu, 27 Feb 2020 19:48:31 GMT
Date: Thu, 27 Feb 2020 19:48:31 GMT
Pragma: no-cache
X-Content-Type-Options: nosniff
X-FRAME-OPTIONS: SAMEORIGIN
X-XSS-Protection: 1; mode=block
Content-Type: application/json;charset=utf-8
Transfer-Encoding: chunked
Server: Jetty(9.4.26.v20200117)
{"FileStatuses":{"FileStatus":[{"accessTime":1582832796457,"blockSize":134217728,"childrenNum":0,"fileId":16419,"group":"supergroup","length":8589934592,"modificationTime":1582832901239,"owner":"systest","pathSuffix":"","permission":"644","replication":2,"storagePolicy":0,"type":"FILE"}]}}
{code}
> Can not handle 8GB file when using webhdfs
> ------------------------------------------
>
> Key: KNOX-2139
> URL: https://issues.apache.org/jira/browse/KNOX-2139
> Project: Apache Knox
> Issue Type: Bug
> Components: Server
> Affects Versions: 1.1.0, 1.2.0
> Reporter: Sean Chow
> Assignee: Kevin Risden
> Priority: Critical
> Attachments: cdp-proxy-api.xml
>
>
> I use knox with webhdfs for a long time, andI uprade my knox version from 0.8
> to 1.2 recent days. It's really strange that knox can't handle file with size
> *8589934592 bytes* when I upload my splited file to hdfs.
> It's easy to reproduce and both knox1.1 and 1.2 have this issue. But is works
> fine in knox0.8.
> Any error log found in gateway.log? No, all logs is clean. From the client
> side (curl), I saw the the url is redirected correctly and failed with
> {{curl: (55) Send failure: Connection reset by peer}} or {{curl: (55) Send
> failure: Broken pipe}}
> I'm sure my network is ok. Any files with other size(smaller or larger) can
> be upload successfully.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
