That isn't malformed AFAICT - that is a redirect to the datanode which we subsquently resolve to the actual endpoint in the rewrite entries that follow it in the log. @Ashok - if you would like to provide an example program that does this, I would try and reproduce when I get a chance.
On Thu, Mar 1, 2018 at 9:25 AM, Sandeep Moré <[email protected]> wrote: > Hello Ashok, > > What version of Knox are you using ? I remember we had a bug with older > Knox version where it would fail to properly rewrite when there was a > special character in the url. > > Looking at your failed url > https://<IP Address>:8443/gateway/default/ > *webhdfs/data/v1/webhdfs/v1/user/* > <https://169.55.247.230:8443/gateway/default/webhdfs/data/v1/webhdfs/v1/user/clsadmin/test17/thread34_46.txt?_=AAAACAAAABAAAADwym5KTMcDaSWnUImQMyca186758MnkmX8hJmrNhyJ-5aFTi-WHglgqcTg5g9LDKyJHZPChwSXP3j9WTVlqvWp_jZv6aUTKh6YVAhBxTJ4MEoUiYA_02ni5Hy8m6ZHU3kHanwLDvDIQ8cO02fYOy-oMqOyRP4l9Poc1bA3Lu8nu3juMddNE7Mzy4EbgzM2uGuFRSqwUZ6OfOXA0nw1xw2lhen6r8F-63s_oC02yywOcVyHF-kes9HVe9S6BNpAQznEXOU5u7M1Imi7jhlXasG75YtbrfPbhrPnVlBE8M8a3-RTVTnZ9_fgAcJsn7U5-Dq07wXWkLRcfF4g6gQwDygHaWJ-j9dt66dt> > > It appears to be malformed. Let me know if that's the case. > > Best, > Sandeep > > On Thu, Mar 1, 2018 at 6:39 AM, Ashok V Jose <[email protected]> wrote: > >> Hi Knox, >> >> We have this program which creates 100 users and they keep running and >> keeps creating different files to hdfs with these commands. >> url='https://<ip address>:8443/gateway/default/ >> webhdfs/v1/user/<username>/test17/' + filetag + '_' + str(num) + >> '.txt?op=CREATE&overwrite=true' >> req=requests.put(url,verify=False,auth=('<username>','<passw >> ord>'),allow_redirects=False,headers=headers) >> >> >> We see that randomly some of the files are not getting created and we >> dont see any network issues or configuration issues. >> >> >> Here is the snapshot this error message in /var/log/knox/gateway.log when >> the file thread85_49.txt is not getting created >> >> 2018-02-28 06:51:28,752 DEBUG realm.AuthenticatingRealm >> (AuthenticatingRealm.java:cacheAuthenticationInfoIfPossible(507)) - >> AuthenticationInfo caching is disabled for info . Submitted token: >> [org.apache.shiro.authc.UsernamePasswordToken - <user Name>, >> rememberMe=false (<IP Address>)]. >> 2018-02-28 06:51:28,752 DEBUG authc.AbstractAuthenticator >> (AbstractAuthenticator.java:authenticate(233)) - Authentication >> successful for token [org.apache.shiro.authc.UsernamePasswordToken - >> <user Name>, rememberMe=false (<IP Address>)]. Returned account >> 2018-02-28 06:51:28,752 DEBUG support.DefaultSubjectContext >> (DefaultSubjectContext.java:resolveSecurityManager(102)) - No >> SecurityManager available in subject context map. Falling back to >> SecurityUtils.getSecurityManager() lookup. >> 2018-02-28 06:51:28,752 DEBUG support.DefaultSubjectContext >> (DefaultSubjectContext.java:resolveSecurityManager(102)) - No >> SecurityManager available in subject context map. Falling back to >> SecurityUtils.getSecurityManager() lookup. >> 2018-02-28 06:51:28,750 ERROR hadoop.gateway >> (UrlRewriteProcessor.java:rewrite(169)) - Failed to rewrite URL: https://<IP >> Address>:8443/gateway/default/webhdfs/data/v1/webhdfs/v1/user/<user >> Name>/test17/thread34_46.txt?_=AAAACAAAABAAAADwym5KTMcDaSWnU >> ImQMyca186758MnkmX8hJmrNhyJ-5aFTi-WHglgqcTg5g9LDKyJHZPChwSXP >> 3j9WTVlqvWp_jZv6aUTKh6YVAhBxTJ4MEoUiYA_02ni5Hy8m6ZHU3kHanwLD >> vDIQ8cO02fYOy-oMqOyRP4l9Poc1bA3Lu8nu3juMddNE7Mzy4EbgzM2uGuFR >> SqwUZ6OfOXA0nw1xw2lhen6r8F-63s_oC02yywOcVyHF-kes9HVe9S6BN >> pAQznEXOU5u7M1Imi7jhlXasG75YtbrfPbhrPnVlBE8M8a3-RTVTnZ9_ >> fgAcJsn7U5-Dq07wXWkLRcfF4g6gQwDygHaWJ-j9dt66dt >> <https://169.55.247.230:8443/gateway/default/webhdfs/data/v1/webhdfs/v1/user/clsadmin/test17/thread34_46.txt?_=AAAACAAAABAAAADwym5KTMcDaSWnUImQMyca186758MnkmX8hJmrNhyJ-5aFTi-WHglgqcTg5g9LDKyJHZPChwSXP3j9WTVlqvWp_jZv6aUTKh6YVAhBxTJ4MEoUiYA_02ni5Hy8m6ZHU3kHanwLDvDIQ8cO02fYOy-oMqOyRP4l9Poc1bA3Lu8nu3juMddNE7Mzy4EbgzM2uGuFRSqwUZ6OfOXA0nw1xw2lhen6r8F-63s_oC02yywOcVyHF-kes9HVe9S6BNpAQznEXOU5u7M1Imi7jhlXasG75YtbrfPbhrPnVlBE8M8a3-RTVTnZ9_fgAcJsn7U5-Dq07wXWkLRcfF4g6gQwDygHaWJ-j9dt66dt>, >> direction: IN via rule: WEBHDFS/webhdfs/inbound/datanode, status: FAILURE >> 2018-02-28 06:51:28,746 DEBUG hadoop.gateway >> (UrlRewriteProcessor.java:rewrite(166)) - Rewrote URL: https://<IP >> Address>:8443/gateway/default/webhdfs/v1/user/<user >> Name>/test17/thread85_49.txt?op=CREATE&overwrite=true >> <https://169.55.247.230:8443/gateway/default/webhdfs/v1/user/clsadmin/test17/thread85_49.txt?op=CREATE&overwrite=true>, >> direction: IN via explicit rule: WEBHDFS/webhdfs/inbound/namenode/file >> to URL: http://<IP Address>:50070/webhdfs/v1/user/<user >> Name>/test17/thread85_49.txt?op=CREATE&overwrite=true >> <http://chs-ymj-017-mn002.bi.services.us-south.bluemix.net:50070/webhdfs/v1/user/clsadmin/test17/thread85_49.txt?op=CREATE&overwrite=true> >> 2018-02-28 06:51:28,745 DEBUG ssl.SslConnection >> (SslConnection.java:fill(526)) - >> SslConnection@166efb61{NEED_UNWRAP,eio=0/-1,di=-1} >> -> HttpConnection@264c413e[FILLING,DecryptedEndPoint@2bf37c3a{/<IP >> Address>:39888<->8443,Open,in,out,-,-,350/300000,HttpConnect >> ion}->SelectChannelEndPoint@7e35950c{/<IP Address>:39888<->8443,Open,in, >> out,-,-,12/300000,SslConnection}{io=0,kio=0,kro=1}][p=HttpParser{s=START,0 >> of >> 0},g=HttpGenerator{s=START},c=HttpChannelOverHttp@7575b295{r=0,c=false,a=IDLE,uri=}] >> unwrap Status = BUFFER_UNDERFLOW HandshakeStatus = NEED_UNWRAP >> >> >> We had authentication set with ldap like this >> >> param> >> <name>main.ldapRealm</name> >> <value>org.apache.hadoop.gateway.shirorealm.KnoxLdapRealm</value> >> </param> >> <param> >> <name>main.ldapContextFactory</name> >> <value>org.apache.hadoop.gateway.shirorealm.KnoxLdapContextF >> actory</value> >> </param> >> <param> >> <name>main.ldapRealm.contextFactory</name> >> <value>$ldapContextFactory</value> >> </param> >> <param> >> >> >> >> Would someone know why are we getting these errors ? >> We dont see this issue as a configuration issue and we dont see network >> load issue here. We see this randomly when running this concurrently. >> >> >> Thanks & Regards, >> Ashok Jose >> >> ********************************************************** >> >> BigInsights Quality >> EGL C Block, 6th Floor >> IBM India Software Labs, >> Mail-Id : [email protected] >> >> >> >
