I am little new to Java. I am working on web scraping program for school. I found out before you can web scrape, you have to create a cookie with userid/password information. I am using HTTPClient library to do that. Following is my code. I am having problem with httpclient. execute(httpget) line, it just goes to error.jsp page, without even going to catch section Please help.
my call: setAuthenticationCookie("","","http://www.msn.com "); public void setAuthenticationCookie( String sUsername, String sPassword, String sURL) { try{ DefaultHttpClient client = new DefaultHttpClient(); // set per default //client.getParams().setParameter( // HttpClientParams.COOKIE_POLICY , CookiePolicy.BROWSER_COMPATIBILITY); HttpGet httpget = new HttpGet(sURL); HttpResponse response = client.execute(httpget); //PROBLEM HERE and above too. sounds like client is not getting a valid handler (however client does not have 0x000 pointer , it does return 213 some number, i guess that means it points to valid address, coming from C++ background, java confuses me) HttpEntity entity = response.getEntity(); if( entity != null){ entity.consumeContent(); } Cookie[] cookies = client.getState ().getCookies(); HttpPost httppost = new HttpPost(sURL); NameValuePair[] valuePair = new NameValuePair[] { new BasicNameValuePair("IDToken1", "username"), new BasicNameValuePair("IDToken2", "password"), }; //Only for https //httppost.setEntity(new UrlEncodedFormEntity(valuePair, HTTP.UTF_8)); response = client.execute(httppost); entity = response.getEntity(); if( entity != null){ entity.consumeContent(); } } catch (Exception e) { e.printStackTrace(System.err); } }