Hi Knox,
We have this program which creates 100 users and they keep running and
keeps creating different files to hdfs with these commands.
url='https://<ip
address>:8443/gateway/default/webhdfs/v1/user/<username>/test17/' + filetag
+ '_' + str(num) + '.txt?op=CREATE&overwrite=true'
req=requests.put(url,verify=False,auth=
('<username>','<password>'),allow_redirects=False,headers=headers)
We see that randomly some of the files are not getting created and we dont
see any network issues or configuration issues.
Here is the snapshot this error message in /var/log/knox/gateway.log when
the file thread85_49.txt is not getting created
2018-02-28 06:51:28,752 DEBUG realm.AuthenticatingRealm
(AuthenticatingRealm.java:cacheAuthenticationInfoIfPossible(507)) -
AuthenticationInfo caching is disabled for info [<user Name>]. Submitted
token: [org.apache.shiro.authc.UsernamePasswordToken - <user Name>,
rememberMe=false (<IP Address>)].
2018-02-28 06:51:28,752 DEBUG authc.AbstractAuthenticator
(AbstractAuthenticator.java:authenticate(233)) - Authentication successful
for token [org.apache.shiro.authc.UsernamePasswordToken - <user Name>,
rememberMe=false (<IP Address>)]. Returned account [<user Name>]
2018-02-28 06:51:28,752 DEBUG support.DefaultSubjectContext
(DefaultSubjectContext.java:resolveSecurityManager(102)) - No
SecurityManager available in subject context map. Falling back to
SecurityUtils.getSecurityManager() lookup.
2018-02-28 06:51:28,752 DEBUG support.DefaultSubjectContext
(DefaultSubjectContext.java:resolveSecurityManager(102)) - No
SecurityManager available in subject context map. Falling back to
SecurityUtils.getSecurityManager() lookup.
2018-02-28 06:51:28,750 ERROR hadoop.gateway
(UrlRewriteProcessor.java:rewrite(169)) - Failed to rewrite URL:
https://<IP
Address>:8443/gateway/default/webhdfs/data/v1/webhdfs/v1/user/<user
Name>/test17/thread34_46.txt?_=AAAACAAAABAAAADwym5KTMcDaSWnUImQMyca186758MnkmX8hJmrNhyJ-5aFTi-WHglgqcTg5g9LDKyJHZPChwSXP3j9WTVlqvWp_jZv6aUTKh6YVAhBxTJ4MEoUiYA_02ni5Hy8m6ZHU3kHanwLDvDIQ8cO02fYOy-oMqOyRP4l9Poc1bA3Lu8nu3juMddNE7Mzy4EbgzM2uGuFRSqwUZ6OfOXA0nw1xw2lhen6r8F-63s_oC02yywOcVyHF-kes9HVe9S6BNpAQznEXOU5u7M1Imi7jhlXasG75YtbrfPbhrPnVlBE8M8a3-RTVTnZ9_fgAcJsn7U5-Dq07wXWkLRcfF4g6gQwDygHaWJ-j9dt66dt
, direction: IN via rule: WEBHDFS/webhdfs/inbound/datanode, status: FAILURE
2018-02-28 06:51:28,746 DEBUG hadoop.gateway
(UrlRewriteProcessor.java:rewrite(166)) - Rewrote URL: https://<IP
Address>:8443/gateway/default/webhdfs/v1/user/<user
Name>/test17/thread85_49.txt?op=CREATE&overwrite=true, direction: IN via
explicit rule: WEBHDFS/webhdfs/inbound/namenode/file to URL: http://<IP
Address>:50070/webhdfs/v1/user/<user
Name>/test17/thread85_49.txt?op=CREATE&overwrite=true
2018-02-28 06:51:28,745 DEBUG ssl.SslConnection (SslConnection.java:fill
(526)) - SslConnection@166efb61{NEED_UNWRAP,eio=0/-1,di=-1} ->
HttpConnection@264c413e[FILLING,DecryptedEndPoint@2bf37c3a{/<IP
Address>:39888<->8443,Open,in,out,-,-,350/300000,HttpConnection}->
SelectChannelEndPoint@7e35950c{/<IP Address>:39888<->
8443,Open,in,out,-,-,12/300000,SslConnection}{io=0,kio=0,kro=1}][p=HttpParser
{s=START,0 of 0},g=HttpGenerator{s=START},c=HttpChannelOverHttp@7575b295
{r=0,c=false,a=IDLE,uri=}] unwrap Status = BUFFER_UNDERFLOW HandshakeStatus
= NEED_UNWRAP
We had authentication set with ldap like this
param>
<name>main.ldapRealm</name>
<value>org.apache.hadoop.gateway.shirorealm.KnoxLdapRealm</value>
</param>
<param>
<name>main.ldapContextFactory</name>
<value>org.apache.hadoop.gateway.shirorealm.KnoxLdapContextFactory</value>
</param>
<param>
<name>main.ldapRealm.contextFactory</name>
<value>$ldapContextFactory</value>
</param>
<param>
Would someone know why are we getting these errors ?
We dont see this issue as a configuration issue and we dont see network
load issue here. We see this randomly when running this concurrently.
Thanks & Regards,
Ashok Jose
**********************************************************
BigInsights Quality
EGL C Block, 6th Floor
IBM India Software Labs,
Mail-Id : [email protected]
