Hello Ashok, What version of Knox are you using ? I remember we had a bug with older Knox version where it would fail to properly rewrite when there was a special character in the url.
Looking at your failed url https://<IP Address>:8443/gateway/default/ *webhdfs/data/v1/webhdfs/v1/user/* <https://169.55.247.230:8443/gateway/default/webhdfs/data/v1/webhdfs/v1/user/clsadmin/test17/thread34_46.txt?_=AAAACAAAABAAAADwym5KTMcDaSWnUImQMyca186758MnkmX8hJmrNhyJ-5aFTi-WHglgqcTg5g9LDKyJHZPChwSXP3j9WTVlqvWp_jZv6aUTKh6YVAhBxTJ4MEoUiYA_02ni5Hy8m6ZHU3kHanwLDvDIQ8cO02fYOy-oMqOyRP4l9Poc1bA3Lu8nu3juMddNE7Mzy4EbgzM2uGuFRSqwUZ6OfOXA0nw1xw2lhen6r8F-63s_oC02yywOcVyHF-kes9HVe9S6BNpAQznEXOU5u7M1Imi7jhlXasG75YtbrfPbhrPnVlBE8M8a3-RTVTnZ9_fgAcJsn7U5-Dq07wXWkLRcfF4g6gQwDygHaWJ-j9dt66dt> It appears to be malformed. Let me know if that's the case. Best, Sandeep On Thu, Mar 1, 2018 at 6:39 AM, Ashok V Jose <ashoj...@in.ibm.com> wrote: > Hi Knox, > > We have this program which creates 100 users and they keep running and > keeps creating different files to hdfs with these commands. > url='https://<ip > address>:8443/gateway/default/webhdfs/v1/user/<username>/test17/' > + filetag + '_' + str(num) + '.txt?op=CREATE&overwrite=true' > req=requests.put(url,verify=False,auth=('<username>','< > password>'),allow_redirects=False,headers=headers) > > > We see that randomly some of the files are not getting created and we dont > see any network issues or configuration issues. > > > Here is the snapshot this error message in /var/log/knox/gateway.log when > the file thread85_49.txt is not getting created > > 2018-02-28 06:51:28,752 DEBUG realm.AuthenticatingRealm > (AuthenticatingRealm.java:cacheAuthenticationInfoIfPossible(507)) - > AuthenticationInfo caching is disabled for info . Submitted token: > [org.apache.shiro.authc.UsernamePasswordToken - <user Name>, > rememberMe=false (<IP Address>)]. > 2018-02-28 06:51:28,752 DEBUG authc.AbstractAuthenticator > (AbstractAuthenticator.java:authenticate(233)) - Authentication > successful for token [org.apache.shiro.authc.UsernamePasswordToken - > <user Name>, rememberMe=false (<IP Address>)]. Returned account > 2018-02-28 06:51:28,752 DEBUG support.DefaultSubjectContext > (DefaultSubjectContext.java:resolveSecurityManager(102)) - No > SecurityManager available in subject context map. Falling back to > SecurityUtils.getSecurityManager() lookup. > 2018-02-28 06:51:28,752 DEBUG support.DefaultSubjectContext > (DefaultSubjectContext.java:resolveSecurityManager(102)) - No > SecurityManager available in subject context map. Falling back to > SecurityUtils.getSecurityManager() lookup. > 2018-02-28 06:51:28,750 ERROR hadoop.gateway > (UrlRewriteProcessor.java:rewrite(169)) > - Failed to rewrite URL: https://<IP Address>:8443/gateway/default/ > webhdfs/data/v1/webhdfs/v1/user/<user Name>/test17/thread34_46.txt?_= > AAAACAAAABAAAADwym5KTMcDaSWnUImQMyca186758MnkmX8hJmrNhyJ-5aFTi- > WHglgqcTg5g9LDKyJHZPChwSXP3j9WTVlqvWp_jZv6aUTKh6YVAhBxTJ4MEoUiYA_ > 02ni5Hy8m6ZHU3kHanwLDvDIQ8cO02fYOy-oMqOyRP4l9Poc1bA3Lu8nu3juMddNE > 7Mzy4EbgzM2uGuFRSqwUZ6OfOXA0nw1xw2lhen6r8F-63s_oC02yywOcVyHF- > kes9HVe9S6BNpAQznEXOU5u7M1Imi7jhlXasG75YtbrfPbhrPnVlBE8M8a3- > RTVTnZ9_fgAcJsn7U5-Dq07wXWkLRcfF4g6gQwDygHaWJ-j9dt66dt > <https://169.55.247.230:8443/gateway/default/webhdfs/data/v1/webhdfs/v1/user/clsadmin/test17/thread34_46.txt?_=AAAACAAAABAAAADwym5KTMcDaSWnUImQMyca186758MnkmX8hJmrNhyJ-5aFTi-WHglgqcTg5g9LDKyJHZPChwSXP3j9WTVlqvWp_jZv6aUTKh6YVAhBxTJ4MEoUiYA_02ni5Hy8m6ZHU3kHanwLDvDIQ8cO02fYOy-oMqOyRP4l9Poc1bA3Lu8nu3juMddNE7Mzy4EbgzM2uGuFRSqwUZ6OfOXA0nw1xw2lhen6r8F-63s_oC02yywOcVyHF-kes9HVe9S6BNpAQznEXOU5u7M1Imi7jhlXasG75YtbrfPbhrPnVlBE8M8a3-RTVTnZ9_fgAcJsn7U5-Dq07wXWkLRcfF4g6gQwDygHaWJ-j9dt66dt>, > direction: IN via rule: WEBHDFS/webhdfs/inbound/datanode, status: FAILURE > 2018-02-28 06:51:28,746 DEBUG hadoop.gateway > (UrlRewriteProcessor.java:rewrite(166)) > - Rewrote URL: https://<IP Address>:8443/gateway/default/webhdfs/v1/user/<user > Name>/test17/thread85_49.txt?op=CREATE&overwrite=true > <https://169.55.247.230:8443/gateway/default/webhdfs/v1/user/clsadmin/test17/thread85_49.txt?op=CREATE&overwrite=true>, > direction: IN via explicit rule: WEBHDFS/webhdfs/inbound/namenode/file to > URL: http://<IP Address>:50070/webhdfs/v1/user/<user > Name>/test17/thread85_49.txt?op=CREATE&overwrite=true > <http://chs-ymj-017-mn002.bi.services.us-south.bluemix.net:50070/webhdfs/v1/user/clsadmin/test17/thread85_49.txt?op=CREATE&overwrite=true> > 2018-02-28 06:51:28,745 DEBUG ssl.SslConnection > (SslConnection.java:fill(526)) - > SslConnection@166efb61{NEED_UNWRAP,eio=0/-1,di=-1} > -> HttpConnection@264c413e[FILLING,DecryptedEndPoint@2bf37c3a{/<IP > Address>:39888<->8443,Open,in,out,-,-,350/300000,HttpConnection}-> > SelectChannelEndPoint@7e35950c{/<IP Address>:39888<->8443,Open,in, > out,-,-,12/300000,SslConnection}{io=0,kio=0,kro=1}][p=HttpParser{s=START,0 > of > 0},g=HttpGenerator{s=START},c=HttpChannelOverHttp@7575b295{r=0,c=false,a=IDLE,uri=}] > unwrap Status = BUFFER_UNDERFLOW HandshakeStatus = NEED_UNWRAP > > > We had authentication set with ldap like this > > param> > <name>main.ldapRealm</name> > <value>org.apache.hadoop.gateway.shirorealm.KnoxLdapRealm</value> > </param> > <param> > <name>main.ldapContextFactory</name> > <value>org.apache.hadoop.gateway.shirorealm.KnoxLdapContextFactory</value> > </param> > <param> > <name>main.ldapRealm.contextFactory</name> > <value>$ldapContextFactory</value> > </param> > <param> > > > > Would someone know why are we getting these errors ? > We dont see this issue as a configuration issue and we dont see network > load issue here. We see this randomly when running this concurrently. > > > Thanks & Regards, > Ashok Jose > > ********************************************************** > > BigInsights Quality > EGL C Block, 6th Floor > IBM India Software Labs, > Mail-Id : ashoj...@in.ibm.com > > >
