Hi guys, recently I have been developing a small to medium sized
application which logs into a database website that contains text files
from various sorces (e.g. The Wall Street Journal), searches for
articles meeting a certain criteria, and then downloads the text of all
the articles which the search yields. Since discovering HttpClient the
project has been going very well, and I have now got to the stage where
I have logged in, searched for the articles, and I am facing a large
list of all the articles meeting the search criteria. Unfortunately I am
having problems on the final POST which results in the server returning
the article page. The POST method is returning a status code 200, but
instead of returning the article it is returning a small page saying “We
are unable to process your request at this time. Please try again in a
few minutes”.
I have been using HTTP Analyzer to monitor exactly what data is sent to
the HTTP server from a) iexplorer, and b) my program. Both programs
matched on all header fields, but they were both posting different data
to the server. Below I have shown the data that is being posted by both
applications:
Iexplorer:
_XFORMSESSSTATE=ABR7MDp7VToiL2hhL2RlZmF1bHQuYXNweCIsMDp7MjoibWJtMCIsM6IBLTB4Ag0iUyIsMToiZW4iLDY6MCw3nAQFd2Fyd2ljazFtBTBsBwI4OjAsOXgDkAB2ATM6YAMDMCw3NTowxAcDMCw0OjAsfAEONzA6Miw3MzoxLDcyOjIsNDFsBAc4OjIsNjk6MSw2egQ2NXkBNngBjA4LMSwwOjI1LDE6MX0sNjB8BZAGeQw1cQ41cg41N2AMfRQ1eQw1YQk1iQQyYANlBjVtBzRxBTR5BjRxBTR5BjRxBTR5BjRxBTR5BjRxBTN9BDNlBjNhCDOMAmwXAzMyOjEsM2UGM20HM3UIM30JMnEFMnkGMnEFMnkGMnEKMnEKMmUGMn0EOXkBOWEDOWkEOXEFN2oJNzdoI3AfAjcxOjEscgMxOGABZQQxfQwxfQcxZQk4fQc4ZQk4dQM4YQY4dQM4fQQ4eQk4nARwEgg4OTowfSwxOnsxMGoEMTBmAzEwkARcC3UvN58WOjAsaQowfBBsBHQFcBJkBX0BN2wN5DgBMSwxMWUFNG0PNmAVKx8HMSw2eQc2fBEp6AZ1JzZxDDlsBzUQBwUxLDU0OjEsNX0tNWUGNXAhkjgxMHkVOHACKz4HMSz%2bOTEs/zkxLDRkCC0%2bBzEweg8xMGYoMTBiCzEwfhoxMGgVK7QHWCF0BD%2b3BzEsMnAg9z0xLDJ4F%2bs%2bMSwyZQs5bBaJPjdxMjmMAmokOTV8LW0sN20CN20CN30SOXwEYQExdTMxlAFhGjFlCTFlDjGAF3QG60MxLDFtDzh9BDhoLOpDMSzrQzEsOGUOOGk2OHUpOW0POY4MMDhobQg0OjEsNTowfX19fREAADwEAAA%3d&sri=True&elks=True&hls=T|article
N|fmt O|c T|report N|fmt O|c O|, T|file N|fmt O|c O|, T|the O|%2b T|j
N|rst O|c T|nnam N|ns O|c T|nrmf N|ns O|c O|, T|nrgn N|ns O|c O|, O|-
O|%2b T|en N|la O|c O|%2b N|pd D|-0008 D| O|d
O|%2b&dfd=FULL&istphst=True&arc=2&ari=1&hdl=[%7B2%3A%7B2%3A%22distdoc%3Aarchive/ArchiveDoc%3A%3AArticle/J000000020060902e2920002a%22%2C0%3A%5B%7B3%3A14406%2C0%3A%22NewsArticle%22%2C4%3A1%2C5%3A%22distdoc%3Aarchive/ArchiveDoc%3A%3AArticle/J000000020060902e2920002a%22%7D%5D%2C1%3A%22Article%22%7D%2C3%3A1%2C0%3A%22J000000020060902e2920002a%22%2C1%3A%22%20Emergency%20Room%3A%20How%20Bristol-Myers%20Fumbled%20Defense%20Of%20%244%20Billion%20Drug%20---%20CEO%20Dolan%27s%20Plan%20to%20Delay%20Generic%20Went%20Haywire%3B%20Board%20to%20Discuss%20His%20Fate%20---%20Doing%20a%20Deal%20With%20the%20%27Devil%27%20%20%22%2C4%3A%22j%22%2C5%3A0%7D]&
My program:
_XFORMSESSSTATE=ABR7MDp7VToiL2hhL2RlZmF1bHQuYXNweCIsMDp7MjoibWJtMCIsM6IBLTB4Ag0iUyIsMToiZW4iLDY6MCw3nAQFd2Fyd2ljazFtBTBsBwI4OjAsOXgDkAB2ATM6YAMDMCw3NTowxAcDMCw0OjAsfAEONzA6Miw3MzoxLDcyOjIsNDFsBAc4OjIsNjk6MSw2egQ2NXkBNngBjA4LMSwwOjI1LDE6MX0sNjB8BZAGeQw1cQ41cg41N2AMfRQ1eQw1YQk1iQQyYANlBjVtBzRxBTR5BjRxBTR5BjRxBTR5BjRxBTR5BjRxBTN9BDNlBjNhCDOMAmwXAzMyOjEsM2UGM20HM3UIM30JMnEFMnkGMnEFMnkGMnEKMnEKMmUGMn0EOXkBOWEDOWkEOXEFN2oJNzdoI3AfAjcxOjEscgMxOGABZQQxfQwxfQcxZQk4fQc4ZQk4dQM4YQY4dQM4fQQ4eQk4nARwEgg4OTowfSwxOnsxMGoEMTBmAzEwkARcC3UvN58WOjAsaQowfBBsBHQFcBJkBX0BN2wN5DgBMSwxMWUFNG0PNmAVKx8HMSw2eQc2fBEp6AZ1JzZxDDlsBzUQBwUxLDU0OjEsNX0tNWUGNXAhkjgxMHkVOHACKz4HMSz%2BOTEs%2FzkxLDRkCC0%2BBzEweg8xMGYoMTBiCzEwfhoxMGgVK7QHWCF0BD%2B3BzEsMnAg9z0xLDJ4F%2Bs%2BMSwyZQs5bBaJPjdxMjmMAmokOTV8LW0sN20CN20CN30SOXwEYQExdTMxlAFhGjFlCTFlDjGAF3QG60MxLDFtDzh9BDhoLOpDMSzrQzEsOGUOOGk2OHUpOW0POY4MMDhobQg0OjEsNTowfX19fREAADwEAAA%3D&sri=True&elks=True&hls=T%7Carticle+N%7Cfmt+O%7Cc+T%7Creport+N%7Cfmt+O%7Cc+O%7C%2C+T%7Cfile+N%7Cfmt+O%7Cc+O%7C%2C+T%7Cthe+O%7C%252b+T%7Cj+N%7Crst+O%7Cc+T%7Cnnam+N%7Cns+O%7Cc+T%7Cnrmf+N%7Cns+O%7Cc+O%7C%2C+T%7Cnrgn+N%7Cns+O%7Cc+O%7C%2C+O%7C-+O%7C%2B+T%7Cen+N%7Cla+O%7Cc+O%7C%2B+N%7Cpd+D%7C-0008+D%7C+O%7Cd+O%7C%2B&dfd=FULL&istphst=True&arc=2&ari=1&hdl=%5B%7B2%3A%7B2%3A%22%3Bdistdoc%3Aarchive%2FArchiveDoc%3A%3AArticle%2FJ000000020060902e2920002a%22%3B%2C0%3A%5B%7B3%3A14406%2C0%3A%22%3BNewsArticle%22%3B%2C4%3A1%2C5%3A%22%3Bdistdoc%3Aarchive%2FArchiveDoc%3A%3AArticle%2FJ000000020060902e2920002a%22%3B%7D%5D%2C1%3A%22%3BArticle%22%3B%7D%2C3%3A1%2C0%3A%22%3BJ000000020060902e2920002a%22%3B%2C1%3A%22%3B+Emergency+Room%3A+How+Bristol-Myers+Fumbled+Defense+Of+%244+Billion+Drug+---+CEO+Dolan%27s+Plan+to+Delay+Generic+Went+Haywire%3B+Board+to+Discuss+His+Fate+---+Doing+a+Deal+With+the+%27Devil%27++%22%3B%2C4%3A%22%3Bj%22%3B%2C5%3A0%7D%5D
Shown below is the NameValuePair[] creation code which corresponds to
this POST method:
NameValuePair[] data = {
new NameValuePair("_XFORMSESSSTATE", getXFORMSESSSTATE(response)),
new NameValuePair("sri", "True"),
new NameValuePair("elks", "True"),
new NameValuePair("hls", "T|article N|fmt O|c T|report N|fmt O|c O|,
T|file N|fmt O|c O|, T|the O|%2b T|j N|rst O|c T|nnam N|ns O|c T|nrmf
N|ns O|c O|, T|nrgn N|ns O|c O|, O|- O|+ T|en N|la O|c O|+ N|pd D|-0008
D| O|d O|+"),
new NameValuePair("dfd", "FULL"),
new NameValuePair("istphst", "True"),
new NameValuePair("arc", "2"),
new NameValuePair("ari", "1"),
new NameValuePair("hdl", gethdl(response))
};
The values for XFORMSESSSTATE and hdl both come from the previous web
page returned by the server and their values in ASCII are:
_XFORMSESSSTATE=ABR7MDp7VToiL2hhL2RlZmF1bHQuYXNweCIsMDp7MjoibWJtMCIsM6IBLTB4Ag0iUyIsMToiZW4iLDY6MCw3nAQFd2Fyd2ljazFtBTBsBwI4OjAsOXgDkAB2ATM6YAMDMCw3NTowxAcDMCw0OjAsfAEONzA6Miw3MzoxLDcyOjIsNDFsBAc4OjIsNjk6MSw2egQ2NXkBNngBjA4LMSwwOjI1LDE6MX0sNjB8BZAGeQw1cQ41cg41N2AMfRQ1eQw1YQk1iQQyYANlBjVtBzRxBTR5BjRxBTR5BjRxBTR5BjRxBTR5BjRxBTN9BDNlBjNhCDOMAmwXAzMyOjEsM2UGM20HM3UIM30JMnEFMnkGMnEFMnkGMnEKMnEKMmUGMn0EOXkBOWEDOWkEOXEFN2oJNzdoI3AfAjcxOjEscgMxOGABZQQxfQwxfQcxZQk4fQc4ZQk4dQM4YQY4dQM4fQQ4eQk4nARwEgg4OTowfSwxOnsxMGoEMTBmAzEwkARcC3UvN58WOjAsaQowfBBsBHQFcBJkBX0BN2wN5DgBMSwxMWUFNG0PNmAVKx8HMSw2eQc2fBEp6AZ1JzZxDDlsBzUQBwUxLDU0OjEsNX0tNWUGNXAhkjgxMHkVOHACKz4HMSz+OTEs/zkxLDRkCC0+BzEweg8xMGYoMTBiCzEwfhoxMGgVK7QHWCF0BD+3BzEsMnAg9z0xLDJ4F+s+MSwyZQs5bBaJPjdxMjmMAmokOTV8LW0sN20CN20CN30SOXwEYQExdTMxlAFhGjFlCTFlDjGAF3QG60MxLDFtDzh9BDhoLOpDMSzrQzEsOGUOOGk2OHUpOW0POY4MMDhobQg0OjEsNTowfX19fREAADwEAAA=
hdl=[{2:{2:";distdoc:archive/ArchiveDoc::Article/J000000020060902e2920002a";,0:[{3:14406,0:";NewsArticle";,4:1,5:";distdoc:archive/ArchiveDoc::Article/J000000020060902e2920002a";}],1:";Article";},3:1,0:";J000000020060902e2920002a";,1:";
Emergency Room: How Bristol-Myers Fumbled Defense Of $4 Billion Drug ---
CEO Dolan's Plan to Delay Generic Went Haywire; Board to Discuss His
Fate --- Doing a Deal With the 'Devil' ";,4:";j";,5:0}]
So from what I can see, iexplorer is encoding the POST data in a
slightly different way to send it to the HTTP server, and thus
unfortunately the server is not recognizing my request.
I don’t know if this is relevant or not, but every POST request until
this point had the header: “Content-Type:
application/x-www-form-urlencoded”, but this POST request has the
header: “Content-Type: application/x-www-form-urlencoded; charset=UTF-8”.
So I guess my question is: how do I make httpClient encode the inputs in
the same way as iexplorer, or is this not possible? Does it have
something to do with the field FORM_URL_ENCODED_CONTENT_TYPE in the
PostMethod class?
Thanks for reading, any help or advice would be greatly appreciated.
Yours sincerely,
Simon Pearson
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
