Re: [PHP] cURL issues posting to an end point
On Thu, Oct 4, 2012 at 1:35 PM, Alejandro Michelin Salomon
amichel...@hotmail.com wrote:
Bastien:
-Mensagem original-
De: Bastien Koert [mailto:phps...@gmail.com]
Enviada em: quinta-feira, 4 de outubro de 2012 11:54
Para: PHP-General
Assunto: [PHP] cURL issues posting to an end point
Hi All,
I have a page that receives third party data into my app, xml data via https
post. This works fine and I receive the data as expected. The issue I am
facing is around posting XML data back as a synchronous response to the post
I receive. I am using the following code:
function sendXMLConfirmation($data)
{
/*
* XML Sender/Client.
*/
// Get our XML. You can declare it here or even load a file.
$xml_builder = '?xml version=1.0 encoding=utf-8?
Envelope version=01.00
Sender
Id/
Credential25412/Credential
/Sender
Recipient
Id/
/Recipient
TransactInfo
transactType=response
TransactId'.$hash.'/TransactId
TimeStamp'.date(Y-m-d H:m
).'/TimeStamp
Status
Code200/Code
ShortDescriptionSuccess/ShortDescription
LongDescriptionCANDIDATE Data transfer was a success/LongDescription
/Status
/TransactInfo
Packet
PacketInfo
packetType=response
PacketId1/PacketId
ActionSET/Action
ManifestManifest
Data/Manifest
Status
Code/
ShortDescription/
LongDescription/
/Status
/PacketInfo
Payload![CDATA[]]/Payload
/Packet
/Envelope
';
// We send XML via CURL using POST with a http header of text/xml.
$ch = curl_init();
$url =
'https://staging.somesite.com/Sprocessor/DispatchMessage.asmx';
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false); // for https
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_TIMEOUT, 4);
curl_setopt($ch, CURLOPT_POSTFIELDS, $xml_builder);
- curl_setopt($ch, CURLOPT_HTTPHEADER, array('Content-Type: text/xml'));
- curl_setopt($ch, CURLOPT_HTTPHEADER, array('Connection: close'));
--- NEW
$sPost = '/Sprocessor/DispatchMessage.asmx';
$sHost = ' https://staging.somesite.com';
curl_setopt($ch, CURLOPT_HTTPHEADER,
array(
'User-Agent: My Program',
'Accept-Encoding: gzip, deflate',
'POST ' . $sPost . ' HTTP/1.1',
'Host: ' . $sHost,
'Content-type: application/soap+xml; charset=utf-8',
'Content-Length: ' . strlen($xml_builder)
);
END NEW -
//Execute the request and also time the transaction ( optional )
$start = array_sum(explode(' ', microtime()));
$result = curl_exec($ch);
curl_close($ch);
// Print CURL result.
echo $ch_result;
}
The endpoint recipient says they are not receiving the data and I am at a
loss to figure out why. Any insight would be appreciated.
Thanks,
--
Bastien
Cat, the other other white meat
--
PHP General Mailing List (http://www.php.net/) To unsubscribe, visit:
http://www.php.net/unsub.php
Thanks, Alejandro,
So much closer that before, however I am getting an error on this.
Unexpected ';' on line 151 which is the closing array );
curl_setopt($ch, CURLOPT_HTTPHEADER,
array(
'User-Agent: My Program',
'Accept-Encoding: gzip, deflate',
'POST ' . $sPost . ' HTTP/1.1',
'Host: ' . $sHost,
'Content-type: application/soap+xml; charset=utf-8',
'Content-Length: ' . strlen($xml_builder)
);
--
Bastien
Cat, the other other white meat
--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] CURL -d
On Mon, 2012-03-26 at 09:45 -0600, QI.VOLMAR QI wrote:
I have this lines:
curl -H Content-Type: application/json -d hello world \
http://api.pusherapp.com/apps/17331/channels/test_channel/events?\
The option -d is for data. But How I can set it on the PHP CURL extension?
I have found that if I set something like array('Hello Word', 'name'
= 'my_name'), for the POST data may occurs. Can you give me a little
help?
Have a look at curl_setopt() which can set those flags as you need
inside PHP:
http://uk3.php.net/manual/en/function.curl-setopt.php
--
Thanks,
Ash
http://www.ashleysheridan.co.uk
Re: [PHP] CURL -d
I know guess that it is a POST field, but the detail is on simulate the -d
without a label. I've already looked at setopt man page, but nothing seems
like what I need.
2012/3/26 Ashley Sheridan a...@ashleysheridan.co.uk
**
On Mon, 2012-03-26 at 09:45 -0600, QI.VOLMAR QI wrote:
I have this lines:
curl -H Content-Type: application/json -d hello world \
http://api.pusherapp.com/apps/17331/channels/test_channel/events?\
http://api.pusherapp.com/apps/17331/channels/test_channel/events?%5C%3E
The option -d is for data. But How I can set it on the PHP CURL extension?
I have found that if I set something like array('Hello Word', 'name'
= 'my_name'), for the POST data may occurs. Can you give me a little
help?
Have a look at curl_setopt() which can set those flags as you need inside
PHP:
http://uk3.php.net/manual/en/function.curl-setopt.php
--
Thanks,
Ash
http://www.ashleysheridan.co.uk
Re: [PHP] curl equivalent in PHP
Thanks Marc. But that need to add the DOM parser to the server. What I am looking for something like iframe in html and that doesn't require any additional PHP modules ( I do would like to avoid additions to the current php; that is why I didn't compiled in curl ) On Fri, Mar 2, 2012 at 7:58 PM, Marc Guay marc.g...@gmail.com wrote: http://simplehtmldom.sourceforge.net/ -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php -- Regards Nibin. http://TechsWare.in
Re: [PHP] curl equivalent in PHP
Thanks Marc. But that need to add the DOM parser to the server. What I am looking for something like iframe in html and that doesn't require any additional PHP modules ( I do would like to avoid additions to the current php; that is why I didn't compiled in curl ) On Fri, Mar 2, 2012 at 8:10 PM, Nibin V M nibi...@gmail.com wrote: Thanks Marc. But that need to add the DOM parser to the server. What I am looking for something like iframe in html and that doesn't require any additional PHP modules ( I do would like to avoid additions to the current php; that is why I didn't compiled in curl ) On Fri, Mar 2, 2012 at 7:58 PM, Marc Guay marc.g...@gmail.com wrote: http://simplehtmldom.sourceforge.net/ -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php -- Regards Nibin. http://TechsWare.in -- Regards Nibin. http://TechsWare.in
Re: [PHP] curl equivalent in PHP
http://www.php.net/manual/en/function.stream-context-create.php On Fri, Mar 2, 2012 at 4:44 PM, Nibin V M nibi...@gmail.com wrote: Thanks Marc. But that need to add the DOM parser to the server. What I am looking for something like iframe in html and that doesn't require any additional PHP modules ( I do would like to avoid additions to the current php; that is why I didn't compiled in curl ) On Fri, Mar 2, 2012 at 8:10 PM, Nibin V M nibi...@gmail.com wrote: Thanks Marc. But that need to add the DOM parser to the server. What I am looking for something like iframe in html and that doesn't require any additional PHP modules ( I do would like to avoid additions to the current php; that is why I didn't compiled in curl ) On Fri, Mar 2, 2012 at 7:58 PM, Marc Guay marc.g...@gmail.com wrote: http://simplehtmldom.sourceforge.net/ -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php -- Regards Nibin. http://TechsWare.in -- Regards Nibin. http://TechsWare.in
Re: [PHP] curl equivalent in PHP
On 03/02/2012 06:26 AM, Nibin V M wrote: Hello, I am trying to display the website content through a php code ( my own websites; doesn't cause copy right issues ). I use curl to display the page via the following simple code. ?php $curl = curl_init(); curl_setopt ($curl, CURLOPT_URL, http://mytest.com;); curl_exec ($curl); curl_close ($curl); ? But on some of my servers, curl isn't enabled! Is there any equivalent code to achieve the same? Thank you, As long as you do not need to perform posts to the other website via the PHP request, you could include, file_get_contents, fopen + fread, etc... All you need to make sure is that allow_url_fopen is enabled. -- Jim Lucas http://www.cmsws.com/ http://www.cmsws.com/examples/ http://www.bendsource.com/ -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] curl equivalent in PHP
Thanks guys :) Now another problem...I use a test domain ( say blahblah.com ) for testing this, which isn't registered yet. So with the given code the index page is loading fine. But when I try to click on any links, it will redirect to the original domain which isn't exists! ( I have actually pointed the test domain to my server via hosts file on the server ) So I wanted to browse the website via a PHP script, same as if I use a browser on the server. Hope you get me! :) Is there any way to achieve this via PHP? What I wanted to On Fri, Mar 2, 2012 at 9:37 PM, Jim Lucas li...@cmsws.com wrote: On 03/02/2012 06:26 AM, Nibin V M wrote: Hello, I am trying to display the website content through a php code ( my own websites; doesn't cause copy right issues ). I use curl to display the page via the following simple code. ?php $curl = curl_init(); curl_setopt ($curl, CURLOPT_URL, http://mytest.com;); curl_exec ($curl); curl_close ($curl); ? But on some of my servers, curl isn't enabled! Is there any equivalent code to achieve the same? Thank you, As long as you do not need to perform posts to the other website via the PHP request, you could include, file_get_contents, fopen + fread, etc... All you need to make sure is that allow_url_fopen is enabled. -- Jim Lucas http://www.cmsws.com/ http://www.cmsws.com/examples/ http://www.bendsource.com/ -- Regards Nibin. http://TechsWare.in
Re: [PHP] curl equivalent in PHP
Hello,
On Fri, Mar 2, 2012 at 8:07 AM, Jim Lucas li...@cmsws.com wrote:
But on some of my servers, curl isn't enabled! Is there any equivalent
code
to achieve the same?
I've used a combination of output buffering [1], readfile() [2] and
caching (specific to the framework I was using).
Simply, you could resort to:
?php readfile('http://www.google.com'); ?
Ofc, the links would need to be absolute in order for the assets to load.
Good luck!
Cheers,
Micky
[1] php.net/ob-start
[2] php.net/readfile
--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] curl equivalent in PHP
On Fri, Mar 2, 2012 at 11:29, Nibin V M nibi...@gmail.com wrote:
Thanks guys :)
Now another problem...I use a test domain ( say blahblah.com ) for testing
this, which isn't registered yet. So with the given code the index page is
loading fine. But when I try to click on any links, it will redirect to the
original domain which isn't exists!
?php
echo
str_replace('blahblah.com','localhost',file_get_contents('remote-domain.example.com'));
?
Modify it as needed, and if you want to surf the site using the
script as a proxy, add the logic to handle that.
--
/Daniel P. Brown
Network Infrastructure Manager
http://www.php.net/
--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] curl equivalent in PHP
Hmm..I am a php newbie ( just started learning it )...
what my need is to display website from my server always for a
non-registered domain.This is the code I use now to display the website
?php
$opts = array(
'http'=array(
'method'=GET,
'header'=Accept-language: en\r\n .
Cookie: foo=bar\r\n
)
);
$context = stream_context_create($opts);
$fp = fopen('http://www.blahblah.com', 'r', false, $context);
fpassthru($fp);
fclose($fp);
?
Of course blahblah.com isn't registered yet. This will load the index page
of blahblah.com fine ( since I use hosts file to resolve blahblah.com on
the server ). But if we click on any link, it will give server not found
error since my local machine is trying to resolve blahblah.com this time.
I need to avoid it and want to load my domain always from the server when
I click any link on the webpage, etc .
How can I modify the above code to make it work! If possible, I request
somebody to tell me the exact code that I need to change in the above
script - since most of the php part is still greek to me :)
Thank you,
On Fri, Mar 2, 2012 at 10:09 PM, Micky Hulse rgmi...@gmail.com wrote:
Hello,
On Fri, Mar 2, 2012 at 8:07 AM, Jim Lucas li...@cmsws.com wrote:
But on some of my servers, curl isn't enabled! Is there any equivalent
code
to achieve the same?
I've used a combination of output buffering [1], readfile() [2] and
caching (specific to the framework I was using).
Simply, you could resort to:
?php readfile('http://www.google.com'); ?
Ofc, the links would need to be absolute in order for the assets to load.
Good luck!
Cheers,
Micky
[1] php.net/ob-start
[2] php.net/readfile
--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php
--
Regards
Nibin.
http://TechsWare.in
Re: [PHP] curl equivalent in PHP
On Fri, Mar 2, 2012 at 6:09 PM, Nibin V M nibi...@gmail.com wrote:
Hmm..I am a php newbie ( just started learning it )...
what my need is to display website from my server always for a
non-registered domain.This is the code I use now to display the website
?php
$opts = array(
'http'=array(
'method'=GET,
'header'=Accept-language: en\r\n .
Cookie: foo=bar\r\n
)
);
$context = stream_context_create($opts);
$fp = fopen('http://www.blahblah.com', 'r', false, $context);
fpassthru($fp);
fclose($fp);
?
Of course blahblah.com isn't registered yet. This will load the index page
of blahblah.com fine ( since I use hosts file to resolve blahblah.com on
the server ). But if we click on any link, it will give server not found
error since my local machine is trying to resolve blahblah.com this time.
I need to avoid it and want to load my domain always from the server when
I click any link on the webpage, etc .
How can I modify the above code to make it work! If possible, I request
somebody to tell me the exact code that I need to change in the above
script - since most of the php part is still greek to me :)
Thank you,
Have you actually read/tried Daniel Browns reply?
The following seems to be all you need...
?php
echo str_replace('blahblah.com','localhost',file_get_contents('blahblah.com'));
?
--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] Curl problems
On Thu, Jan 12, 2012 at 12:20 AM, Rick Dwyer rpdw...@earthlink.net wrote: Hello all. I use curl to make a call to another page on my site... but it operates erroneously sometimes working... sometimes not. The page it calls creates an email and I can see on the server the email in the queue when it's working. If I echo out the URL the curl command is supposed to load and load it manually, it works without fail. Any help on what I am doing wrong below is greatly appreciated. Thanks. $curl_handle=curl_init(); curl_setopt($curl_handle,CURLOPT_URL,'https://mydomain.com/email_confirmation.htm?id_order='.$id_order.'sess_id='.$sess_id); curl_exec($curl_handle); curl_close($curl_handle); --Rick It's maybe not a real answer to your question, but if all you want to do is call that page, why don't you just use file_get_contents(https://mydomain.com/email_confirmation.htm?id_order='.$id_order.'sess_id='.$sess_id); (See [1]) It works out of the box, and I have found curl unstable too sometimes. Matijn [1] www.php.net/file_get_contents -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] Curl problems
On Jan 11, 2012, at 6:29 PM, Matijn Woudt wrote: On Thu, Jan 12, 2012 at 12:20 AM, Rick Dwyer rpdw...@earthlink.net wrote: Hello all. I use curl to make a call to another page on my site... but it operates erroneously sometimes working... sometimes not. The page it calls creates an email and I can see on the server the email in the queue when it's working. If I echo out the URL the curl command is supposed to load and load it manually, it works without fail. Any help on what I am doing wrong below is greatly appreciated. Thanks. $curl_handle=curl_init(); curl_setopt($curl_handle,CURLOPT_URL,'https://mydomain.com/email_confirmation.htm?id_order='.$id_order.'sess_id='.$sess_id) ; curl_exec($curl_handle); curl_close($curl_handle); --Rick It's maybe not a real answer to your question, but if all you want to do is call that page, why don't you just use file_get_contents(https://mydomain.com/email_confirmation.htm?id_order='.$id_order.'sess_id='.$sess_id ); (See [1]) It works out of the box, and I have found curl unstable too sometimes. Matijn Thanks Matijn, But I get Notice: file_get_contents() [function.file-get-contents]: Unable to find the wrapper https - did you forget to enable it when you configured PHP?... I'm using a hosting provider and I don't believe they will enable this for security reasons. --Rick -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] Curl problems
On Thu, Jan 12, 2012 at 12:44 AM, Rick Dwyer rpdw...@earthlink.net wrote: On Jan 11, 2012, at 6:29 PM, Matijn Woudt wrote: On Thu, Jan 12, 2012 at 12:20 AM, Rick Dwyer rpdw...@earthlink.net wrote: Hello all. I use curl to make a call to another page on my site... but it operates erroneously sometimes working... sometimes not. The page it calls creates an email and I can see on the server the email in the queue when it's working. If I echo out the URL the curl command is supposed to load and load it manually, it works without fail. Any help on what I am doing wrong below is greatly appreciated. Thanks. $curl_handle=curl_init(); curl_setopt($curl_handle,CURLOPT_URL,'https://mydomain.com/email_confirmation.htm?id_order='.$id_order.'sess_id='.$sess_id); curl_exec($curl_handle); curl_close($curl_handle); --Rick It's maybe not a real answer to your question, but if all you want to do is call that page, why don't you just use file_get_contents(https://mydomain.com/email_confirmation.htm?id_order='.$id_order.'sess_id='.$sess_id); (See [1]) It works out of the box, and I have found curl unstable too sometimes. Matijn Thanks Matijn, But I get Notice: file_get_contents() [function.file-get-contents]: Unable to find the wrapper https - did you forget to enable it when you configured PHP?... I'm using a hosting provider and I don't believe they will enable this for security reasons. --Rick It seems like they have not compiled PHP with SSL support, or they're using a pretty old version. Anyway, you're probably stuck with cURL then, check the return of curl_exec, and if false, call curl_error to get an error message. Matijn -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] Curl cost
On 9/28/2011 1:28 AM, muad shibani wrote: what are the costs of using PHP Curl to show another websites on my site as stumbleon do .. traffic, memory or what? That isn't how stumbleupon does it. You might want to take a closer look at the HTML to see how they do what they do. -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] Curl cost
On Wed, Sep 28, 2011 at 11:54, Jim Lucas li...@cmsws.com wrote: That isn't how stumbleupon does it. You might want to take a closer look at the HTML to see how they do what they do. He said stumbleON, actually. Looks like they simply aggregate some of your personal social networking data into a single-sign-on dashboard presentation. Certainly no service I'd use, but I suppose for some it has its merits. -- /Daniel P. Brown Network Infrastructure Manager http://www.php.net/ -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] Curl cost
On 9/28/2011 9:05 AM, Daniel Brown wrote: On Wed, Sep 28, 2011 at 11:54, Jim Lucas li...@cmsws.com wrote: That isn't how stumbleupon does it. You might want to take a closer look at the HTML to see how they do what they do. He said stumbleON, actually. Looks like they simply aggregate some of your personal social networking data into a single-sign-on dashboard presentation. Certainly no service I'd use, but I suppose for some it has its merits. I saw that, but @ss-umed it was a typo. My bad. -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] Curl cost
On Wed, Sep 28, 2011 at 14:13, Jim Lucas li...@cmsws.com wrote: I saw that, but @ss-umed it was a typo. My bad. I'd presumed the same at first. You're in good company still. -- /Daniel P. Brown Network Infrastructure Manager http://www.php.net/ -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] Curl cost
On Wed, Sep 28, 2011 at 4:06 PM, tamouse mailing lists tamouse.li...@gmail.com wrote: On Wed, Sep 28, 2011 at 3:28 AM, muad shibani muad.shib...@gmail.com wrote: what are the costs of using PHP Curl to show another websites on my site as stumbleon do .. traffic, memory or what? If you use curl to suck the web page into a variable, could be tremendous. Better to curl it into a temporary file if you're going that way. Just a followup -- if you want to show other websites on your own page, a friendly way to do this is using an iframe: ?php $site=http://www.example.com;; ? h2a href=?php echo $site??php echo $site?/a/h2 iframe src=?php echo $site? frameborder=0 title=?php echo $site? width=80% height=200pxspan style=color:red;font-weight:bold;Your browser does not support iframes/span/iframe That way, the site gets the hit, and you aren't copying anything. If you do go this way, make sure to provide a break out so the user can open the target site in a new window/tab. -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] curl rtmp
On Sun, Jan 9, 2011 at 2:58 PM, Tontonq Tontonq root...@gmail.com wrote: does cUrl supports rtmp protocol? if so is there any example? These are obvious by searching for the terms, which seem to be quite specific to have not found an answer in the search engines. do we need enable different library? so if not can we save rtmp by curl? if not is there any other rtmp downloader that u know ? You seem to know enough to have answered this by yourself, almost in your own questions. -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] curl and variable parameters in hyperlink
foreach ($_POST as $key=$post) {
$post=str_replace( , +, $post);
$url.=$key.=.$post.;
}
Hi Bob,
One thing I see is that you're appending values on to the end of a url that
already has 2 values, so you should place the at the beginning of your
line instead of at the end. Value 2 is being merged with whatever value is
coming after it in your example. So, try something like:
$url .= . $key. = .$post;
Also, I'd suggest url encoding the post value rather than merely hunting
down spaces and swapping them out with +.
Hope this helps,
Adam
--
Nephtali: PHP web framework that functions beautifully
http://nephtaliproject.com
Re: [PHP] curl and variable parameters in hyperlink
On Wednesday 24 November 2010,
Bob Keightley bob.keight...@virgin.net wrote:
I already have a curl script that gets the web page, but it doesn't pass
the parameters
Hello Bob,
Being new to PHP I haven't the first idea how to modify it so that it
does.
Script is as follows:
$url = http://www.xx.com/query.asp?param1=val1param2=val2;;
foreach ($_POST as $key=$post) {
$post=str_replace( , +, $post);
$url.=$key.=.$post.;
}
Instead of concatenating strings, I suggest to use the http_build_query()
function:
?php
$data = array('param1' = 'val1',
'param2' = 'val2',
'param3' = 'val3');
$query = http_build_query($data);
echo Query: {$query}\n;
?
So, to create a query string based on your $_POST request parameters,
you just need to use the function as follows:
?php
$query = http_build_query($_POST);
echo Query: {$query}\n;
?
This will create the proper query string to be used on your URL.
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 0);
$data = curl_exec($ch);
curl_close($ch);
$data=str_replace('.asp', '.php', $data);
echo $data;
This returns the web page, but ignores val1 and val2 which are necessary
to execute the query.
Best regards,
--
Daniel Molina Wegener dmw [at] coder [dot] cl
System Programmer Web Developer
Phone: +56 (2) 979-0277 | Blog: http://coder.cl/
signature.asc
Description: This is a digitally signed message part.
Re: [PHP] curl and variable parameters in hyperlink
Guess I've not explained this very well. The external page I am fetching using another curl script has asp query hyperlinks in it. I do not know the variable names in each query or the values they have. All I want to do is replace the external url referenced in those links with a url which is on my site (so that I can change style sheet, image locations etc.) but the parameters to the original asp query link have to be captured and passed to my php script for it to return data. I know there must be a way of doing this as there's a site which is referencing the same third party site and doing the same thing, and no, they won't tell me how! -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
RE: [PHP] curl and variable parameters in hyperlink
-Original Message- From: Bob Keightley [mailto:bob.keight...@virgin.net] Sent: Wednesday, November 24, 2010 11:25 AM To: php-general@lists.php.net Subject: Re: [PHP] curl and variable parameters in hyperlink Guess I've not explained this very well. The external page I am fetching using another curl script has asp query hyperlinks in it. I do not know the variable names in each query or the values they have. All I want to do is replace the external url referenced in those links with a url which is on my site (so that I can change style sheet, image locations etc.) but the parameters to the original asp query link have to be captured and passed to my php script for it to return data. I know there must be a way of doing this as there's a site which is referencing the same third party site and doing the same thing, and no, they won't tell me how! Bob, There's several issues with this. 1) They're providing the info which you're trying to pass it as your own and most likely that they won't get any positive results, as some company do analysis on visits to improve the marketing strategy, customer service, customer relations, etc. You're just skewing their data analysis and prevent them from becoming successful. Which may force them to go out of business then you'll lose your source of information. 2) Waste of bandwidth. 3rd party you user instead of just you user. 3) I'm don't like that fact that some people take others work and try to pass it as their own. I think many folks on this list feels the same way. So why not just quote them and give them the credit they deserve. That's probably the reason why you're not able to find much info on how to do this :) Regards, Tommy -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
RE: [PHP] curl and variable parameters in hyperlink
Thanks to all for your interest so far but as I said, new to PHP. Changed POST to GET and all now working fine! Tommy, the third party site is fully aware of what I am doing. We pay them to use the data, as do lots of other companies in our industry. Supplying data is what they do. We could not use the site at all without being given a valid user name. Far from 'preventing them being successful' the more users they have the more successful they become. We do not 'pass the work as our own' they are credited on our site as suppliers of the content, so there really is no issue there. -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
RE: [PHP] curl and variable parameters in hyperlink
Depending on how the website is setup you can curl or soap parameters. http://www.php.net/manual/en/book.curl.php http://www.php.net/manual/en/book.soap.php Richard L. Buskirk -Original Message- From: Bob Keightley [mailto:bob.keight...@virgin.net] Sent: Tuesday, November 23, 2010 9:12 PM To: php-general@lists.php.net Subject: [PHP] curl and variable parameters in hyperlink First use of Curl so probably a very simple problem - I have a script which loads a third party web page into my site. That page contains links which have variable parameters i.e. www.blahblah.asp?param1=xxxparam2=yyy. Any clues as to what I need to do to pass these parameters in a curl script appreciated. -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
RE: [PHP] curl and variable parameters in hyperlink
I already have a curl script that gets the web page, but it doesn't pass the
parameters
Being new to PHP I haven't the first idea how to modify it so that it does.
Script is as follows:
$url = http://www.xx.com/query.asp?param1=val1param2=val2;;
foreach ($_POST as $key=$post) {
$post=str_replace( , +, $post);
$url.=$key.=.$post.;
}
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 0);
$data = curl_exec($ch);
curl_close($ch);
$data=str_replace('.asp', '.php', $data);
echo $data;
This returns the web page, but ignores val1 and val2 which are necessary to
execute the query.
--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] curl
# yum install php4-curl or # yum install php-curl Shiplu Mokadd.im My talks, http://talk.cmyweb.net Follow me, http://twitter.com/shiplu SUST Programmers, http://groups.google.com/group/p2psust Innovation distinguishes bet ... ... (ask Steve Jobs the rest) -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] curl help or other suggestion
On Thu, 2010-06-17 at 09:05 -0400, Michael Alaimo wrote: I am trying to use register_shutdown_function. Previous to the script shutting down I use curl to grab a website. Basically I want to know if the user has hit the stop button or left the page prematurely. The only problem is that curl runs and finishes its call before the shutdown function is called. Because of this I have no way to know if the user canceled the request somehow. Any suggestions? Mike You can't. When a visitor closes their browser, or stops the page from completing its loading, no message is sent to the server, so your PHP script won't know about it. The only way I know of currently to check for this is use some form of regular check-in, with Ajax or something, that calls home at regular intervals to inform your script that it is indeed still connected. If it fails to call home in time it either means there's lag on the connection or they broke it. Thanks, Ash http://www.ashleysheridan.co.uk
Re: [PHP] curl help or other suggestion
On Thu, 2010-06-17 at 09:37 -0400, Michael Alaimo wrote: On Thu, 2010-06-17 at 09:05 -0400, Michael Alaimo wrote: I am trying to use register_shutdown_function. Previous to the script shutting down I use curl to grab a website. Basically I want to know if the user has hit the stop button or left the page prematurely. The only problem is that curl runs and finishes its call before the shutdown function is called. Because of this I have no way to know if the user canceled the request somehow. Any suggestions? Mike You can't. When a visitor closes their browser, or stops the page from completing its loading, no message is sent to the server, so your PHP script won't know about it. The only way I know of currently to check for this is use some form of regular check-in, with Ajax or something, that calls home at regular intervals to inform your script that it is indeed still connected. If it fails to call home in time it either means there's lag on the connection or they broke it. Thanks, Ash http://www.ashleysheridan.co.uk I wish that curl would just stop processing. I think you can use connection tracking to know what the user has done. I just cannot use it with my current situation as the code is. Mike If this situation is occurring often, maybe a re-think of the structure of your app. What is cURL fetching? Is it something that you can cache at your end, like an RSS feed, etc? If so, maybe use the local cache and only refresh if the local copy is too old. Thanks, Ash http://www.ashleysheridan.co.uk
Re: [PHP] curl help or other suggestion
On Thu, Jun 17, 2010 at 9:05 AM, Michael Alaimo mala...@sesda2.com wrote: I am trying to use register_shutdown_function. Previous to the script shutting down I use curl to grab a website. Basically I want to know if the user has hit the stop button or left the page prematurely. The only problem is that curl runs and finishes its call before the shutdown function is called. Because of this I have no way to know if the user canceled the request somehow. Any suggestions? Mike I don't know that it will affect cURL, but if you are just looking to log completed versus abandoned requests, this might help: http://www.php.net/manual/en/function.ignore-user-abort.php Andrew -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] curl help or other suggestion
As it turns out curl_multi does not block. I was able to use that plus connection tracking to solve my problem. Thanks Ash and Andrew. Mike On Thu, Jun 17, 2010 at 9:05 AM, Michael Alaimo mala...@sesda2.com wrote: I am trying to use register_shutdown_function. Â Previous to the script shutting down I use curl to grab a website. Basically I want to know if the user has hit the stop button or left the page prematurely. The only problem is that curl runs and finishes its call before the shutdown function is called. Â Because of this I Â have no way to know if the user canceled the request somehow. Â Any suggestions? Mike I don't know that it will affect cURL, but if you are just looking to log completed versus abandoned requests, this might help: http://www.php.net/manual/en/function.ignore-user-abort.php Andrew -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] CURL cannot connect to URL - IP address - after successful connection
On 4/25/10, ioan...@btinternet.com wrote: I can return a target page - once, but then on refresh within a few hours the script curl_error is that it cannot connect to the host and return is empty. Failed to connect to host is a pretty strange error if they're doing anything regarding cookies and so on, IMO - I think I'd expect at least a connection to be established before they decide they don't like you. Have you used curl's --trace --trace-ascii options? -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] CURL cannot connect to URL - IP address - after successful connection
On 2010/04/29 19:46, Gary . wrote: On 4/25/10, ioan...@btinternet.com wrote: I can return a target page - once, but then on refresh within a few hours the script curl_error is that it cannot connect to the host and return is empty. Failed to connect to host is a pretty strange error if they're doing anything regarding cookies and so on, IMO - I think I'd expect at least a connection to be established before they decide they don't like you. Have you used curl's --trace --trace-ascii options? Is that debug_backtrace() in php, as I am not using the command line (can't work out how to get the window up having downloaded curl, I am not up to 'building libraries' that seems to be needed). debug_backtrace() does not give any useful information other than saying the target link fails to connect (this is after it connects once, and then on refresh and for several hours does not connect). I guess there is some program that notes the calling IP address and if it is in a range it does not like, adds it to a list and refuses subsequent connections to the same address for a while. Cookies are not required when using the browser directly. John -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] CURL cannot connect to URL - IP address - after successful connection
On 4/29/10, ioan...@btinternet.com wrote: On 2010/04/29 19:46, Gary . wrote: Failed to connect to host is a pretty strange error if they're doing anything regarding cookies and so on, IMO - I think I'd expect at least a connection to be established before they decide they don't like you. Have you used curl's --trace --trace-ascii options? Is that debug_backtrace() in php Not sure :-P as I am not using the command line (can't work out how to get the window up having downloaded curl, I am not up to 'building libraries' that seems to be needed). Windows? http://curl.haxx.se/download.html *n*x variants should allow installing via their packge management systems. -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] CURL cannot connect to URL - IP address - after successful connection
I think the answer is: ISPs have a different range of addresses from host providers, so it is possible to block requests from host servers, so from scripts. John -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
RE: [PHP] CURL cannot connect to URL - IP address - after successful connection
-Original Message- From: ioan...@btinternet.com [mailto:ioan...@btinternet.com] Sent: Wednesday, April 28, 2010 7:03 AM To: 'PHP' Subject: Re: [PHP] CURL cannot connect to URL - IP address - after successful connection I think the answer is: ISPs have a different range of addresses from host providers, so it is possible to block requests from host servers, so from scripts. John That's possible but very unlikely in your case. Since you were to able to get the necessary information on the 1st request, but failed on subsequent means it has an anti-bot mechanism in place. Bypassing anti-bot requires in-depth reverse-engineering of the targeted site. Regards, Tommy -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
RE: [PHP] CURL cannot connect to URL - IP address - after successful connection
-Original Message- From: ioan...@btinternet.com [mailto:ioan...@btinternet.com] Sent: Sunday, April 25, 2010 6:18 AM To: php-general@lists.php.net Subject: [PHP] CURL cannot connect to URL - IP address - after successful connection I can return a target page - once, but then on refresh within a few hours the script curl_error is that it cannot connect to the host and return is empty. The target URL is an ip address, not a named url, so maybe it has something to do with DNS. I am on a shared server. Any ideas on why this happens? John Just to eliminate all possibilities, are you to open the same URL/URI in the web pages repeatedly? Also, what happens when you fake the user agent in the web browser? The target site may have some anti bot mechanism in place to reduce stress/load on the server(s). Regards, Tommy -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
RE: [PHP] CURL cannot connect to URL - IP address - after successful connection
-Original Message- From: Tommy Pham [mailto:tommy...@gmail.com] Sent: Monday, April 26, 2010 1:59 AM To: 'php-general@lists.php.net' Subject: RE: [PHP] CURL cannot connect to URL - IP address - after successful connection -Original Message- From: ioan...@btinternet.com [mailto:ioan...@btinternet.com] Sent: Sunday, April 25, 2010 6:18 AM To: php-general@lists.php.net Subject: [PHP] CURL cannot connect to URL - IP address - after successful connection I can return a target page - once, but then on refresh within a few hours the script curl_error is that it cannot connect to the host and return is empty. The target URL is an ip address, not a named url, so maybe it has something to do with DNS. I am on a shared server. Any ideas on why this happens? John Just to eliminate all possibilities, are you to open the same URL/URI in the web pages repeatedly? Also, what happens when you fake the user agent in the web browser? The target site may have some anti bot mechanism in place to reduce stress/load on the server(s). Regards, Tommy One more thing, check it with cookies enabled/disabled in the web browser too. -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] CURL cannot connect to URL - IP address - after successful connection
Just to eliminate all possibilities, are you to open the same URL/URI in
the
web pages repeatedly? Also, what happens when you fake the user agent in
the web browser? The target site may have some anti bot mechanism in
place to reduce stress/load on the server(s).
Regards,
Tommy
One more thing, check it with cookies enabled/disabled in the web browser
too.
Having deleted cookies on the browser and disabled them, it still does
not like various user agents:
$useragent = array('Mozilla','Opera','Microsoft Internet
Explorer','ia_archiver');
$os = array('Windows','Windows XP','Linux','Windows NT','Windows
2000','OSX');
//random user agent code
$agent = $useragent[rand(0,3)].'/'.rand(1,8).'.'.rand(0,9).'
('.$os[rand(0,5)].' '.rand(1,7).'.'.rand(0,9).'; en-US;)';
//would give something like Mozilla/3.5 (Windows 5.4; en-US;)
-- OR --
//$useragent='Google Image - Googlebot-Image/1.0 (
http://www.googlebot.com/bot.html)';
//$useragent=MSN Live - msnbot-Products/1.0
(+http://search.msn.com/msnbot.htm);
-- OR --
//$agent = DocZilla/1.0 (Windows; U; WinNT4.0; en-US; rv:1.0.0)
Gecko/20020804;
I am just calling the page manually, once at a time. It is probable
that there is some anti-bot measures. Page would probably not want to
be indexed as it is providing ever changing content. How to use this
for normal level of use for real user just in a different site?
John
--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] CURL cannot connect to URL - IP address - after successful connection
On Mon, 2010-04-26 at 12:05 +0900, ioan...@btinternet.com wrote:
Just to eliminate all possibilities, are you to open the same URL/URI in
the
web pages repeatedly? Also, what happens when you fake the user agent in
the web browser? The target site may have some anti bot mechanism in
place to reduce stress/load on the server(s).
Regards,
Tommy
One more thing, check it with cookies enabled/disabled in the web browser
too.
Having deleted cookies on the browser and disabled them, it still does
not like various user agents:
$useragent = array('Mozilla','Opera','Microsoft Internet
Explorer','ia_archiver');
$os = array('Windows','Windows XP','Linux','Windows NT','Windows
2000','OSX');
//random user agent code
$agent = $useragent[rand(0,3)].'/'.rand(1,8).'.'.rand(0,9).'
('.$os[rand(0,5)].' '.rand(1,7).'.'.rand(0,9).'; en-US;)';
//would give something like Mozilla/3.5 (Windows 5.4; en-US;)
-- OR --
//$useragent='Google Image - Googlebot-Image/1.0 (
http://www.googlebot.com/bot.html)';
//$useragent=MSN Live - msnbot-Products/1.0
(+http://search.msn.com/msnbot.htm);
-- OR --
//$agent = DocZilla/1.0 (Windows; U; WinNT4.0; en-US; rv:1.0.0)
Gecko/20020804;
I am just calling the page manually, once at a time. It is probable
that there is some anti-bot measures. Page would probably not want to
be indexed as it is providing ever changing content. How to use this
for normal level of use for real user just in a different site?
John
How frequently do you request the page? Maybe playing about with that
would resolve it? Is it possible to randomise the request frequency a
bit?
Thanks,
Ash
http://www.ashleysheridan.co.uk
Re: [PHP] CURL cannot connect to URL - IP address - after successful connection
On 2010/04/26 20:01, Ashley Sheridan wrote: How frequently do you request the page? Maybe playing about with that would resolve it? Is it possible to randomise the request frequency a bit? Thanks, Ash http://www.ashleysheridan.co.uk Just manually for testing, and it would be used for human requests. Say occasionally 5, 10, 30 minutes intervals etc. There must be other parameters that are being passed so that the site can determine that the request is coming from the same user and through a script request, because it works normally from the browser so just refusing a second call from the same IP address (which could be a browser with static or unchanged IP address) is not what is happening. It must be determining that it is through a server from another site via curl or similar. John -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
RE: [PHP] CURL cannot connect to URL - IP address - after successful connection
-Original Message- From: ioan...@btinternet.com [mailto:ioan...@btinternet.com] Sent: Sunday, April 25, 2010 10:44 PM To: a...@ashleysheridan.co.uk; tommy...@gmail.com Tommy Pham Subject: Re: [PHP] CURL cannot connect to URL - IP address - after successful connection The answer I got from support desk on my shared server: 'You are trying to curl to a datapipe server, if it is rejecting the server name and port, you will need to take that up with them.' John I assume that you did full testing with the browser as I suggested? If everything works, one other thing to keep in mind is that the target also may implement reverse DNS lookup in their anti-bot. One good way to test that is to remote in via SSH (if on Linux/Unix) to test with wget. Otherwise, I'm pretty sure that target site have some anti-bot mechanisms in place. Microseconds of analyzing valid 'user' requests is better than processing 2-3 seconds and sending the response which will consume bandwidth. What you could also try is setting different user-agents for every request and use cookies in cURL should the target site have an anti-bot mechanism. Regards, Tommy -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
RE: [PHP] CURL cannot connect to URL - IP address - after successful connection
-Original Message- From: ioan...@btinternet.com [mailto:ioan...@btinternet.com] Sent: Monday, April 26, 2010 7:10 AM To: Tommy Pham Subject: Re: [PHP] CURL cannot connect to URL - IP address - after successful connection On 2010/04/27 1:13, Tommy Pham wrote: -Original Message- I assume that you did full testing with the browser as I suggested? If everything works, one other thing to keep in mind is that the target also may implement reverse DNS lookup in their anti-bot. One good way to test that is to remote in via SSH (if on Linux/Unix) to test with wget. Otherwise, I'm pretty sure that target site have some anti-bot mechanisms in place. Microseconds of analyzing valid 'user' requests is better than processing 2-3 seconds and sending the response which will consume bandwidth. What you could also try is setting different user-agents for every request and use cookies in cURL should the target site have an anti-bot mechanism. Regards, Tommy Yes, I think I have tested with/without cookies on the browser, trying different user agents (code emailed previously using array and rand) and cookies are used in script/not used. And it works on subsequent requests? I cannot work out how to use Putty/ssh/public private keys etc..wget... Learning how to use that is easier than learning to code PHP, IMO. I read about some problems with curl setting the port and a required patch on the server. John If cURL requires some kind of patch as you say, then it wouldn't have work in the first place. Perhaps it's better to post your (obfuscated personal data) code. Or try on your local PC on your local web server to eliminate possibility of proxies, anti-bots, etc... problems to test that your code works as intended and not cURL problem as you say. I didn't have problems using cURL before. But then my targeted sites were very big companies and didn't care about bots much. Regards, Tommy PS: Always reply to the list so others in the future can benefit unless it's something personal. -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] CURL cannot connect to URL - IP address - after successful connection
On Sun, 2010-04-25 at 22:17 +0900, ioan...@btinternet.com wrote: I can return a target page - once, but then on refresh within a few hours the script curl_error is that it cannot connect to the host and return is empty. The target URL is an ip address, not a named url, so maybe it has something to do with DNS. I am on a shared server. Any ideas on why this happens? John No, DNS is a Domain Named Server used to turn a domain name into an IP address. As you say you're using an IP address directly, it won't go near DNS. Are there any messages in the logs that would give more specific information? Thanks, Ash http://www.ashleysheridan.co.uk
Re: [PHP] CURL cannot connect to URL - IP address - after successful connection
This is all I see in the error log: SUEXEC error_log: [2010-04-25 16:45:42]: uid: (1116/myname) gid: (1118/myname) cmd: fcgiwrapper John -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] cURL on freebsd
On Tue, Mar 30, 2010 at 7:36 PM, Shane Hill shanehil...@gmail.com wrote: can anyone tell me why the cURL code below produces a POST request as expected on linux, but the same code on freebsd is forced to be a GET request? the linux box is running php 5.2.11 the freebsd box is running 5.3.2 is there a bug in 5.3.2? a search does not turn anything up. Just curious, what version of FreeBSD? Did you compile PHP extensions or used the packages? -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] cURL on freebsd
On Mar 30, 2010, at 19:50, Tommy Pham tommy...@gmail.com wrote: On Tue, Mar 30, 2010 at 7:36 PM, Shane Hill shanehil...@gmail.com wrote: can anyone tell me why the cURL code below produces a POST request as expected on linux, but the same code on freebsd is forced to be a GET request? the linux box is running php 5.2.11 the freebsd box is running 5.3.2 is there a bug in 5.3.2? a search does not turn anything up. Just curious, what version of FreeBSD? Did you compile PHP extensions or used the packages? Freebsd 7.2 stable I compiled everything enohPi ym morf tnes -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] Curl output
using the base tag is a solution to your problem if the links are relative.. http://www.w3schools.com/TAGS/tag_base.asp but I am not sure if base tag works outside html/html try... base href=URL/base before curl_init() where URL is parsed using PHP's parse_url() function on http://example.com in either case if there are any external js/css files cURL wont fetch external files(and if they are internal/inline your browser should display them already).. your browser will have to fetch them for you, net console of firebug will be helpful in that case. -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] Curl output
On Fri, 2009-10-02 at 12:51 +0530, kranthi wrote: using the base tag is a solution to your problem if the links are relative.. http://www.w3schools.com/TAGS/tag_base.asp but I am not sure if base tag works outside html/html try... base href=URL/base before curl_init() where URL is parsed using PHP's parse_url() function on http://example.com in either case if there are any external js/css files cURL wont fetch external files(and if they are internal/inline your browser should display them already).. your browser will have to fetch them for you, net console of firebug will be helpful in that case. Some browser security settings may not allow you to run Javascript code that exists on another server though. The base tag can be used to reference the external CSS and Javascript, but watch out for security settings on client agents. Thanks, Ash http://www.ashleysheridan.co.uk -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] Curl output
Some browser security settings may not allow you to run Javascript code that exists on another server though not many users use those kind of browsers, because if they do most of the websites which use CDNs will not work. Firstly, it is not a good idea to fetch an entire web page and snow it to an user. (use iframes if this is a must) Secondly, this idea may not be feasible if the web page in question uses AJAX, because none of the browsers allow cross domain AJAX requests As a side note, use str_replace(head, headbase/base, $text) if base tag doesnot work On 02/10/2009, Ashley Sheridan a...@ashleysheridan.co.uk wrote: On Fri, 2009-10-02 at 12:51 +0530, kranthi wrote: using the base tag is a solution to your problem if the links are relative.. http://www.w3schools.com/TAGS/tag_base.asp but I am not sure if base tag works outside html/html try... base href=URL/base before curl_init() where URL is parsed using PHP's parse_url() function on http://example.com in either case if there are any external js/css files cURL wont fetch external files(and if they are internal/inline your browser should display them already).. your browser will have to fetch them for you, net console of firebug will be helpful in that case. Some browser security settings may not allow you to run Javascript code that exists on another server though. The base tag can be used to reference the external CSS and Javascript, but watch out for security settings on client agents. Thanks, Ash http://www.ashleysheridan.co.uk -- Kranthi. -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] Curl output
On Fri, 2009-10-02 at 15:59 +0530, kranthi wrote: not many users use those kind of browsers, because if they do most of the websites which use CDNs will not work. I've read that the upcoming Firefox 4 may have some features built in for this sort of thing, and there are plugins out there for most browsers that can do this as an added layer of security. On this front though, has anyone else read about Microsofts Gazelle browser/os? I'm not trying to plug it here or anything (I'm an anti Windows person myself!) but the concept looks interesting. It goes further than Chrome where each tab has it's own process by giving each domain specific object its own process/sandbox. Thanks, Ash http://www.ashleysheridan.co.uk -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] Curl output
I've read that the upcoming Firefox 4 may have some features built in for this sort of thing, and there are plugins out there for most browsers that can do this as an added layer of security. Sorry but I could not understand what you meant by this coming back to original problem... you should keep in mind that if base tag is used, the links (a href=...) in that page will become absolute links instead of relative links -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] Curl output
On Fri, 2009-10-02 at 18:07 +0530, kranthi wrote: I've read that the upcoming Firefox 4 may have some features built in for this sort of thing, and there are plugins out there for most browsers that can do this as an added layer of security. Sorry but I could not understand what you meant by this coming back to original problem... you should keep in mind that if base tag is used, the links (a href=...) in that page will become absolute links instead of relative links this is this: Some browser security settings may not allow you to run Javascript code that exists on another server though Also, you won't have to worry about the links, unless you wanted the links that were relative to the remote site to be relative to your own. This would only work out in such a way if you'd scraped their whole site for content, which would beg the question, why would someone honestly need to scrape an entire site? Thanks, Ash http://www.ashleysheridan.co.uk -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] Curl output
On Thu, Oct 01, 2009 at 04:37:14PM -0700, gbhumphrey wrote: Hi, I am doing a basical curl call and can get the webpage I want. However when it prints to the screen, it only prints the text, not css or any javascript calls that run on page load. Is there a way to make it do that? thanks using a basic curl call $curl_handle=curl_init(); curl_setopt($curl_handle,CURLOPT_URL,'http://example.com'); curl_setopt($ch, CURLOPT_RETURNTRANSFER,0); // return into a variable curl_exec($curl_handle); I don't know about the javascript, but if the CSS is an external file, then it wouldn't get dragged along with the original file, and thus wouldn't be there to style anything on the HTML page. Just a guess. Paul -- Paul M. Foster -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] Curl output
gbhumphrey wrote: Hi, I am doing a basical curl call and can get the webpage I want. However when it prints to the screen, it only prints the text, not css or any javascript calls that run on page load. Is there a way to make it do that? thanks using a basic curl call $curl_handle=curl_init(); curl_setopt($curl_handle,CURLOPT_URL,'http://example.com'); curl_setopt($ch, CURLOPT_RETURNTRANSFER,0); // return into a variable curl_exec($curl_handle); Along the lines of what Paul has already alluded to, you will probably find upon further examination of the html source that the linked files (js, css, etc...) are probably using relative paths, not absolute paths including the domain name. To fix this, you would have to do a little source code rewriting to add the full domain name and directory path information to the linked in url structure. I know if you do this with wget (cli app) that you can have it re-write the domain + path url references. But, I'm not sure if cURL has an option for that. You might RTM on the cURL options page to find out. http://php.net/curl -- Jim Lucas Some men are born to greatness, some achieve greatness, and some have greatness thrust upon them. Twelfth Night, Act II, Scene V by William Shakespeare -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] CURL intermittant problem
On Thu, Jun 11, 2009 at 19:07, ioan...@btinternet.comioan...@btinternet.com wrote: The target URL works perfectly and quicly if put directly into the browser URL line. Do you know of any other reasons for problems with such a connection? Is the script being run located on the same system as which you're using the browser to (successfully) connect to the target host? If not, check to see if there's a network issue - a downed or improperly bound interface, SELinux restrictions, blacklisted by the remote machine, etc. - on the server on which the script it hosted. In addition, as simple and obvious as it is, if it's a private network, make sure the machine on which the script is hosted is within the same network as the target machine. -- /Daniel P. Brown daniel.br...@parasane.net || danbr...@php.net http://www.parasane.net/ || http://www.pilotpig.net/ 50% Off All Shared Hosting Plans at PilotPig: Use Coupon DOW1 -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] CURL intermittant problem
It looks like a connection problem, You has to make sure php can connect to the IP ?php echo exec( 'ping 10.20.30.40' ); ? You can remove the IP from the source, just add an entry in the /etc/hosts file Mrtn On Thu, Jun 11, 2009 at 8:14 PM, Daniel Brown danbr...@php.net wrote: On Thu, Jun 11, 2009 at 19:07, ioan...@btinternet.comioan...@btinternet.com wrote: The target URL works perfectly and quicly if put directly into the browser URL line. Do you know of any other reasons for problems with such a connection? Is the script being run located on the same system as which you're using the browser to (successfully) connect to the target host? If not, check to see if there's a network issue - a downed or improperly bound interface, SELinux restrictions, blacklisted by the remote machine, etc. - on the server on which the script it hosted. In addition, as simple and obvious as it is, if it's a private network, make sure the machine on which the script is hosted is within the same network as the target machine. -- /Daniel P. Brown daniel.br...@parasane.net || danbr...@php.net http://www.parasane.net/ || http://www.pilotpig.net/ 50% Off All Shared Hosting Plans at PilotPig: Use Coupon DOW1 -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php -- Martin Scotta
Re: [PHP] cURL loop?
On Thu, May 28, 2009 at 23:31, espontaneo acohln...@gmail.com wrote: Hello! I am currently working on a script that will scrape data from a property advertising web page. The web page has multiple pages. What I'm getting is only the first page. What I wanted to do is to use curl to scrape all the data from that page. I just learned php so I don't know how I can do this. There are a variety of ways, but rather than writing your own spider script, you may want to look into the built-in spidering capabilities of `wget` and `curl` from the command line. Both have Windows and *NIX builds, so platform isn't an issue; so long as you have access to the shell, you should be fine. -- /Daniel P. Brown daniel.br...@parasane.net || danbr...@php.net http://www.parasane.net/ || http://www.pilotpig.net/ 50% Off All Shared Hosting Plans at PilotPig: Use Coupon DOW1 -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] cURL - Error 400
On Fri, Apr 17, 2009 at 7:58 AM, David quick.webmas...@gmail.com wrote:
On Fri, Apr 17, 2009 at 8:55 PM, Andrew Ballard aball...@gmail.com wrote:
On Fri, Apr 17, 2009 at 12:41 AM, David quick.webmas...@gmail.com wrote:
On Thu, Apr 16, 2009 at 11:48 PM, haliphax halip...@gmail.com wrote:
On Wed, Apr 15, 2009 at 9:17 PM, David quick.webmas...@gmail.com
wrote:
Except I also need to POST data to the server to login. After I've
logged
in, I then need to use cookies to maintain a session.
Doing that via file_get_contents() just isn't possible.
Thanks
On Thu, Apr 16, 2009 at 2:30 AM, haliphax halip...@gmail.com wrote:
On Wed, Apr 15, 2009 at 10:36 AM, David quick.webmas...@gmail.com
wrote:
I was wondering if anyone could please help me with this cURL
script
since I
keep getting error 400 from the web server:
http://pastebin.ca/1392840
It worked until around a month ago which is when they presumably
made
changes to the site. Except I can't figure out what configuration
option
in
the cURL PHP script needs to be changed. I can visit the site
perfectly
in
Lynx, Firefox and IE.
Are you just trying to get the contents of the page, or is there
something special you're doing? If it's just the contents you're
after, try file_get_contents() if allow_url_fopen is set to TRUE for
your PHP installation.
http://php.net/file_get_contents
http://php.net/allow_url_fopen
David, please refrain from top-posting.
As for cURL login/session handling... I have an automated script that
connects to a phpBB bulletin board, and here are the settings that
have worked for me:
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_COOKIESESSION, true);
curl_setopt($ch, CURLOPT_HEADER, false);
curl_setopt($ch, CURLOPT_COOKIEFILE, {$homedir}cookiefile);
curl_setopt($ch, CURLOPT_COOKIEJAR, {$homedir}cookiefile);
curl_setopt($ch, CURLOPT_COOKIE, session_name() . '=' . session_id());
I would think CURLOPT_FOLLOWLOCATION and the CURLOPT_COOKIE* options
are most important for resolving your issue.
HTH,
--
// Todd
Hi,
Sorry, that didn't work. The website is still returning error 400 with
CURLOPT_COOKIESESSION and CURLOPT_COOKIE enabled.
I did some experimentation in Firefox by blocking and deleting all
cookies
from the site. When I then visited the site, I was able to reach the
logon
page without returning error 400 so I doubt it's cookies that is the
problem.
I also tried changing the HTTP headers to:
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Connection: Keep-Alive
Content-Type: text/html; charset=UTF-8
Which didn't work either.
Thanks
Does the site require a valid HTTP_REFERER? I haven't seen
CURLOPT_REFERER in any of your examples. (If I missed it somewhere,
just ignore the noise.)
Have you set your CURLOPT_COOKIEFILE properly? Also--you may need to
unlink() it before you log-in using your cURL script to make sure that
any invalid cookie values (i.e., expired session ID) are cleared out.
unlink({$homedir}cookiefile);
curl_setopt($ch, CURLOPT_COOKIESESSION, true);
curl_setopt($ch, CURLOPT_COOKIEFILE, {$homedir}cookiefile);
curl_setopt($ch, CURLOPT_COOKIEJAR, {$homedir}cookiefile);
curl_setopt($ch, CURLOPT_COOKIE, session_name() . '=' . session_id());
...
(where $homedir is /home/haliphax/ or some such value.)
HTH,
--
// Todd
--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] cURL - Error 400
On Mon, Apr 20, 2009 at 9:35 PM, haliphax halip...@gmail.com wrote:
On Fri, Apr 17, 2009 at 7:58 AM, David quick.webmas...@gmail.com wrote:
On Fri, Apr 17, 2009 at 8:55 PM, Andrew Ballard aball...@gmail.com
wrote:
On Fri, Apr 17, 2009 at 12:41 AM, David quick.webmas...@gmail.com
wrote:
On Thu, Apr 16, 2009 at 11:48 PM, haliphax halip...@gmail.com
wrote:
On Wed, Apr 15, 2009 at 9:17 PM, David quick.webmas...@gmail.com
wrote:
Except I also need to POST data to the server to login. After I've
logged
in, I then need to use cookies to maintain a session.
Doing that via file_get_contents() just isn't possible.
Thanks
On Thu, Apr 16, 2009 at 2:30 AM, haliphax halip...@gmail.com
wrote:
On Wed, Apr 15, 2009 at 10:36 AM, David
quick.webmas...@gmail.com
wrote:
I was wondering if anyone could please help me with this cURL
script
since I
keep getting error 400 from the web server:
http://pastebin.ca/1392840
It worked until around a month ago which is when they presumably
made
changes to the site. Except I can't figure out what
configuration
option
in
the cURL PHP script needs to be changed. I can visit the site
perfectly
in
Lynx, Firefox and IE.
Are you just trying to get the contents of the page, or is there
something special you're doing? If it's just the contents you're
after, try file_get_contents() if allow_url_fopen is set to TRUE
for
your PHP installation.
http://php.net/file_get_contents
http://php.net/allow_url_fopen
David, please refrain from top-posting.
As for cURL login/session handling... I have an automated script that
connects to a phpBB bulletin board, and here are the settings that
have worked for me:
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_COOKIESESSION, true);
curl_setopt($ch, CURLOPT_HEADER, false);
curl_setopt($ch, CURLOPT_COOKIEFILE, {$homedir}cookiefile);
curl_setopt($ch, CURLOPT_COOKIEJAR, {$homedir}cookiefile);
curl_setopt($ch, CURLOPT_COOKIE, session_name() . '=' .
session_id());
I would think CURLOPT_FOLLOWLOCATION and the CURLOPT_COOKIE* options
are most important for resolving your issue.
HTH,
--
// Todd
Hi,
Sorry, that didn't work. The website is still returning error 400 with
CURLOPT_COOKIESESSION and CURLOPT_COOKIE enabled.
I did some experimentation in Firefox by blocking and deleting all
cookies
from the site. When I then visited the site, I was able to reach the
logon
page without returning error 400 so I doubt it's cookies that is the
problem.
I also tried changing the HTTP headers to:
Accept:
text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Connection: Keep-Alive
Content-Type: text/html; charset=UTF-8
Which didn't work either.
Thanks
Does the site require a valid HTTP_REFERER? I haven't seen
CURLOPT_REFERER in any of your examples. (If I missed it somewhere,
just ignore the noise.)
Have you set your CURLOPT_COOKIEFILE properly? Also--you may need to
unlink() it before you log-in using your cURL script to make sure that
any invalid cookie values (i.e., expired session ID) are cleared out.
unlink({$homedir}cookiefile);
curl_setopt($ch, CURLOPT_COOKIESESSION, true);
curl_setopt($ch, CURLOPT_COOKIEFILE, {$homedir}cookiefile);
curl_setopt($ch, CURLOPT_COOKIEJAR, {$homedir}cookiefile);
curl_setopt($ch, CURLOPT_COOKIE, session_name() . '=' . session_id());
...
(where $homedir is /home/haliphax/ or some such value.)
HTH,
--
// Todd
Hi,
The cookie file is currently empty but when I delete it, a new one is
created once the script is ran.
Thanks
Re: [PHP] cURL - Error 400
On Thu, Apr 16, 2009 at 11:41 PM, David quick.webmas...@gmail.com wrote:
Hi,
Sorry, that didn't work. The website is still returning error 400 with
CURLOPT_COOKIESESSION and CURLOPT_COOKIE enabled.
I did some experimentation in Firefox by blocking and deleting all cookies
from the site. When I then visited the site, I was able to reach the logon
page without returning error 400 so I doubt it's cookies that is the
problem.
I also tried changing the HTTP headers to:
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Connection: Keep-Alive
Content-Type: text/html; charset=UTF-8
Which didn't work either.
Thanks
On Thu, Apr 16, 2009 at 11:48 PM, haliphax halip...@gmail.com wrote:
On Wed, Apr 15, 2009 at 9:17 PM, David quick.webmas...@gmail.com wrote:
Except I also need to POST data to the server to login. After I've
logged
in, I then need to use cookies to maintain a session.
Doing that via file_get_contents() just isn't possible.
Thanks
On Thu, Apr 16, 2009 at 2:30 AM, haliphax halip...@gmail.com wrote:
On Wed, Apr 15, 2009 at 10:36 AM, David quick.webmas...@gmail.com
wrote:
I was wondering if anyone could please help me with this cURL script
since I
keep getting error 400 from the web server:
http://pastebin.ca/1392840
It worked until around a month ago which is when they presumably made
changes to the site. Except I can't figure out what configuration
option
in
the cURL PHP script needs to be changed. I can visit the site
perfectly
in
Lynx, Firefox and IE.
Are you just trying to get the contents of the page, or is there
something special you're doing? If it's just the contents you're
after, try file_get_contents() if allow_url_fopen is set to TRUE for
your PHP installation.
http://php.net/file_get_contents
http://php.net/allow_url_fopen
David, please refrain from top-posting.
As for cURL login/session handling... I have an automated script that
connects to a phpBB bulletin board, and here are the settings that
have worked for me:
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_COOKIESESSION, true);
curl_setopt($ch, CURLOPT_HEADER, false);
curl_setopt($ch, CURLOPT_COOKIEFILE, {$homedir}cookiefile);
curl_setopt($ch, CURLOPT_COOKIEJAR, {$homedir}cookiefile);
curl_setopt($ch, CURLOPT_COOKIE, session_name() . '=' . session_id());
I would think CURLOPT_FOLLOWLOCATION and the CURLOPT_COOKIE* options
are most important for resolving your issue.
PLEASE.
STOP.
TOP.
POSTING.
Did you try CURLOPT_FOLLOWLOCATION? If your page uses any sort of
redirect, cURL will probably fail without this option.
--
// Todd
--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] cURL - Error 400
On Fri, Apr 17, 2009 at 8:20 PM, haliphax halip...@gmail.com wrote:
On Thu, Apr 16, 2009 at 11:41 PM, David quick.webmas...@gmail.com wrote:
Hi,
Sorry, that didn't work. The website is still returning error 400 with
CURLOPT_COOKIESESSION and CURLOPT_COOKIE enabled.
I did some experimentation in Firefox by blocking and deleting all
cookies
from the site. When I then visited the site, I was able to reach the
logon
page without returning error 400 so I doubt it's cookies that is the
problem.
I also tried changing the HTTP headers to:
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Connection: Keep-Alive
Content-Type: text/html; charset=UTF-8
Which didn't work either.
Thanks
On Thu, Apr 16, 2009 at 11:48 PM, haliphax halip...@gmail.com wrote:
On Wed, Apr 15, 2009 at 9:17 PM, David quick.webmas...@gmail.com
wrote:
Except I also need to POST data to the server to login. After I've
logged
in, I then need to use cookies to maintain a session.
Doing that via file_get_contents() just isn't possible.
Thanks
On Thu, Apr 16, 2009 at 2:30 AM, haliphax halip...@gmail.com wrote:
On Wed, Apr 15, 2009 at 10:36 AM, David quick.webmas...@gmail.com
wrote:
I was wondering if anyone could please help me with this cURL
script
since I
keep getting error 400 from the web server:
http://pastebin.ca/1392840
It worked until around a month ago which is when they presumably
made
changes to the site. Except I can't figure out what configuration
option
in
the cURL PHP script needs to be changed. I can visit the site
perfectly
in
Lynx, Firefox and IE.
Are you just trying to get the contents of the page, or is there
something special you're doing? If it's just the contents you're
after, try file_get_contents() if allow_url_fopen is set to TRUE for
your PHP installation.
http://php.net/file_get_contents
http://php.net/allow_url_fopen
David, please refrain from top-posting.
As for cURL login/session handling... I have an automated script that
connects to a phpBB bulletin board, and here are the settings that
have worked for me:
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_COOKIESESSION, true);
curl_setopt($ch, CURLOPT_HEADER, false);
curl_setopt($ch, CURLOPT_COOKIEFILE, {$homedir}cookiefile);
curl_setopt($ch, CURLOPT_COOKIEJAR, {$homedir}cookiefile);
curl_setopt($ch, CURLOPT_COOKIE, session_name() . '=' . session_id());
I would think CURLOPT_FOLLOWLOCATION and the CURLOPT_COOKIE* options
are most important for resolving your issue.
PLEASE.
STOP.
TOP.
POSTING.
Did you try CURLOPT_FOLLOWLOCATION? If your page uses any sort of
redirect, cURL will probably fail without this option.
--
// Todd
Yes, I've already tried CURLOPT_FOLLOWLOCATION. I believe it was included it
in the code I initially uploaded to pastebin.ca.
Re: [PHP] cURL - Error 400
On Fri, Apr 17, 2009 at 12:41 AM, David quick.webmas...@gmail.com wrote:
On Thu, Apr 16, 2009 at 11:48 PM, haliphax halip...@gmail.com wrote:
On Wed, Apr 15, 2009 at 9:17 PM, David quick.webmas...@gmail.com wrote:
Except I also need to POST data to the server to login. After I've logged
in, I then need to use cookies to maintain a session.
Doing that via file_get_contents() just isn't possible.
Thanks
On Thu, Apr 16, 2009 at 2:30 AM, haliphax halip...@gmail.com wrote:
On Wed, Apr 15, 2009 at 10:36 AM, David quick.webmas...@gmail.com
wrote:
I was wondering if anyone could please help me with this cURL script
since I
keep getting error 400 from the web server:
http://pastebin.ca/1392840
It worked until around a month ago which is when they presumably made
changes to the site. Except I can't figure out what configuration
option
in
the cURL PHP script needs to be changed. I can visit the site
perfectly
in
Lynx, Firefox and IE.
Are you just trying to get the contents of the page, or is there
something special you're doing? If it's just the contents you're
after, try file_get_contents() if allow_url_fopen is set to TRUE for
your PHP installation.
http://php.net/file_get_contents
http://php.net/allow_url_fopen
David, please refrain from top-posting.
As for cURL login/session handling... I have an automated script that
connects to a phpBB bulletin board, and here are the settings that
have worked for me:
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_COOKIESESSION, true);
curl_setopt($ch, CURLOPT_HEADER, false);
curl_setopt($ch, CURLOPT_COOKIEFILE, {$homedir}cookiefile);
curl_setopt($ch, CURLOPT_COOKIEJAR, {$homedir}cookiefile);
curl_setopt($ch, CURLOPT_COOKIE, session_name() . '=' . session_id());
I would think CURLOPT_FOLLOWLOCATION and the CURLOPT_COOKIE* options
are most important for resolving your issue.
HTH,
--
// Todd
Hi,
Sorry, that didn't work. The website is still returning error 400 with
CURLOPT_COOKIESESSION and CURLOPT_COOKIE enabled.
I did some experimentation in Firefox by blocking and deleting all cookies
from the site. When I then visited the site, I was able to reach the logon
page without returning error 400 so I doubt it's cookies that is the
problem.
I also tried changing the HTTP headers to:
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Connection: Keep-Alive
Content-Type: text/html; charset=UTF-8
Which didn't work either.
Thanks
Does the site require a valid HTTP_REFERER? I haven't seen
CURLOPT_REFERER in any of your examples. (If I missed it somewhere,
just ignore the noise.)
Andrew
--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] cURL - Error 400
On Fri, Apr 17, 2009 at 8:55 PM, Andrew Ballard aball...@gmail.com wrote:
On Fri, Apr 17, 2009 at 12:41 AM, David quick.webmas...@gmail.com wrote:
On Thu, Apr 16, 2009 at 11:48 PM, haliphax halip...@gmail.com wrote:
On Wed, Apr 15, 2009 at 9:17 PM, David quick.webmas...@gmail.com
wrote:
Except I also need to POST data to the server to login. After I've
logged
in, I then need to use cookies to maintain a session.
Doing that via file_get_contents() just isn't possible.
Thanks
On Thu, Apr 16, 2009 at 2:30 AM, haliphax halip...@gmail.com wrote:
On Wed, Apr 15, 2009 at 10:36 AM, David quick.webmas...@gmail.com
wrote:
I was wondering if anyone could please help me with this cURL
script
since I
keep getting error 400 from the web server:
http://pastebin.ca/1392840
It worked until around a month ago which is when they presumably
made
changes to the site. Except I can't figure out what configuration
option
in
the cURL PHP script needs to be changed. I can visit the site
perfectly
in
Lynx, Firefox and IE.
Are you just trying to get the contents of the page, or is there
something special you're doing? If it's just the contents you're
after, try file_get_contents() if allow_url_fopen is set to TRUE for
your PHP installation.
http://php.net/file_get_contents
http://php.net/allow_url_fopen
David, please refrain from top-posting.
As for cURL login/session handling... I have an automated script that
connects to a phpBB bulletin board, and here are the settings that
have worked for me:
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_COOKIESESSION, true);
curl_setopt($ch, CURLOPT_HEADER, false);
curl_setopt($ch, CURLOPT_COOKIEFILE, {$homedir}cookiefile);
curl_setopt($ch, CURLOPT_COOKIEJAR, {$homedir}cookiefile);
curl_setopt($ch, CURLOPT_COOKIE, session_name() . '=' . session_id());
I would think CURLOPT_FOLLOWLOCATION and the CURLOPT_COOKIE* options
are most important for resolving your issue.
HTH,
--
// Todd
Hi,
Sorry, that didn't work. The website is still returning error 400 with
CURLOPT_COOKIESESSION and CURLOPT_COOKIE enabled.
I did some experimentation in Firefox by blocking and deleting all
cookies
from the site. When I then visited the site, I was able to reach the
logon
page without returning error 400 so I doubt it's cookies that is the
problem.
I also tried changing the HTTP headers to:
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Connection: Keep-Alive
Content-Type: text/html; charset=UTF-8
Which didn't work either.
Thanks
Does the site require a valid HTTP_REFERER? I haven't seen
CURLOPT_REFERER in any of your examples. (If I missed it somewhere,
just ignore the noise.)
Andrew
Hi,
I'm pretty sure that it doesn't require a referring site.
Thanks
Re: [PHP] cURL Download
2009/4/17 Robbert van Andel robb...@vafam.com: I've been struggling to download a file from a network file share using cURL, or whatever else will work. All I want to do is get the contents of a text file. But when I run the code below I get this error Error: 37 - Couldn't open file \\server\share\test.txt. I'm running my website on a linux box, but the server is on a Windows file server. I've verified that the file share works when I use Windows Explorer and that the server can get the file via the command line using smbclient. Any help would be appreciated. ?PHP $file = file:server\\share\\test.txt; echo pGetting $file/p\n; // create a new cURL resource $ch = curl_init(); // set URL and other appropriate options curl_setopt($ch, CURLOPT_URL, $file); curl_setopt($ch, CURLOPT_USERPWD, username:password); curl_setopt($ch, CURLOPT_HEADER, false); // grab file and pass it to the browser echo pFile: . curl_exec($ch) ./p\n; echo pError: . curl_errno($ch) . - . curl_error($ch) . /p\n; ? Thanks, Robbert Although I can't help you with this I'm pretty sure that cURL does not have smb support which is the protocol that windows uses for this file sharing. Maybe you should try googleling around for something like php smb access. A quick search turned up with this: http://www.phpclasses.org/browse/file/20566.html -- Currently developing a browsergame... http://www.p-game.de Trade - Expand - Fight Follow me on twitter! http://twitter.com/moortier -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] cURL - Error 400
On Wed, Apr 15, 2009 at 9:17 PM, David quick.webmas...@gmail.com wrote:
Except I also need to POST data to the server to login. After I've logged
in, I then need to use cookies to maintain a session.
Doing that via file_get_contents() just isn't possible.
Thanks
On Thu, Apr 16, 2009 at 2:30 AM, haliphax halip...@gmail.com wrote:
On Wed, Apr 15, 2009 at 10:36 AM, David quick.webmas...@gmail.com wrote:
I was wondering if anyone could please help me with this cURL script
since I
keep getting error 400 from the web server:
http://pastebin.ca/1392840
It worked until around a month ago which is when they presumably made
changes to the site. Except I can't figure out what configuration option
in
the cURL PHP script needs to be changed. I can visit the site perfectly
in
Lynx, Firefox and IE.
Are you just trying to get the contents of the page, or is there
something special you're doing? If it's just the contents you're
after, try file_get_contents() if allow_url_fopen is set to TRUE for
your PHP installation.
http://php.net/file_get_contents
http://php.net/allow_url_fopen
David, please refrain from top-posting.
As for cURL login/session handling... I have an automated script that
connects to a phpBB bulletin board, and here are the settings that
have worked for me:
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_COOKIESESSION, true);
curl_setopt($ch, CURLOPT_HEADER, false);
curl_setopt($ch, CURLOPT_COOKIEFILE, {$homedir}cookiefile);
curl_setopt($ch, CURLOPT_COOKIEJAR, {$homedir}cookiefile);
curl_setopt($ch, CURLOPT_COOKIE, session_name() . '=' . session_id());
I would think CURLOPT_FOLLOWLOCATION and the CURLOPT_COOKIE* options
are most important for resolving your issue.
HTH,
--
// Todd
--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] cURL - Error 400
Hi,
Sorry, that didn't work. The website is still returning error 400 with
CURLOPT_COOKIESESSION and CURLOPT_COOKIE enabled.
I did some experimentation in Firefox by blocking and deleting all cookies
from the site. When I then visited the site, I was able to reach the logon
page without returning error 400 so I doubt it's cookies that is the
problem.
I also tried changing the HTTP headers to:
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Connection: Keep-Alive
Content-Type: text/html; charset=UTF-8
Which didn't work either.
Thanks
On Thu, Apr 16, 2009 at 11:48 PM, haliphax halip...@gmail.com wrote:
On Wed, Apr 15, 2009 at 9:17 PM, David quick.webmas...@gmail.com wrote:
Except I also need to POST data to the server to login. After I've logged
in, I then need to use cookies to maintain a session.
Doing that via file_get_contents() just isn't possible.
Thanks
On Thu, Apr 16, 2009 at 2:30 AM, haliphax halip...@gmail.com wrote:
On Wed, Apr 15, 2009 at 10:36 AM, David quick.webmas...@gmail.com
wrote:
I was wondering if anyone could please help me with this cURL script
since I
keep getting error 400 from the web server:
http://pastebin.ca/1392840
It worked until around a month ago which is when they presumably made
changes to the site. Except I can't figure out what configuration
option
in
the cURL PHP script needs to be changed. I can visit the site
perfectly
in
Lynx, Firefox and IE.
Are you just trying to get the contents of the page, or is there
something special you're doing? If it's just the contents you're
after, try file_get_contents() if allow_url_fopen is set to TRUE for
your PHP installation.
http://php.net/file_get_contents
http://php.net/allow_url_fopen
David, please refrain from top-posting.
As for cURL login/session handling... I have an automated script that
connects to a phpBB bulletin board, and here are the settings that
have worked for me:
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_COOKIESESSION, true);
curl_setopt($ch, CURLOPT_HEADER, false);
curl_setopt($ch, CURLOPT_COOKIEFILE, {$homedir}cookiefile);
curl_setopt($ch, CURLOPT_COOKIEJAR, {$homedir}cookiefile);
curl_setopt($ch, CURLOPT_COOKIE, session_name() . '=' . session_id());
I would think CURLOPT_FOLLOWLOCATION and the CURLOPT_COOKIE* options
are most important for resolving your issue.
HTH,
--
// Todd
Re: [PHP] cURL - Error 400
On Wed, Apr 15, 2009 at 10:36 AM, David quick.webmas...@gmail.com wrote: I was wondering if anyone could please help me with this cURL script since I keep getting error 400 from the web server: http://pastebin.ca/1392840 It worked until around a month ago which is when they presumably made changes to the site. Except I can't figure out what configuration option in the cURL PHP script needs to be changed. I can visit the site perfectly in Lynx, Firefox and IE. Are you just trying to get the contents of the page, or is there something special you're doing? If it's just the contents you're after, try file_get_contents() if allow_url_fopen is set to TRUE for your PHP installation. http://php.net/file_get_contents http://php.net/allow_url_fopen Hope this helps, -- // Todd -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] cURL - Error 400
Except I also need to POST data to the server to login. After I've logged in, I then need to use cookies to maintain a session. Doing that via file_get_contents() just isn't possible. Thanks On Thu, Apr 16, 2009 at 2:30 AM, haliphax halip...@gmail.com wrote: On Wed, Apr 15, 2009 at 10:36 AM, David quick.webmas...@gmail.com wrote: I was wondering if anyone could please help me with this cURL script since I keep getting error 400 from the web server: http://pastebin.ca/1392840 It worked until around a month ago which is when they presumably made changes to the site. Except I can't figure out what configuration option in the cURL PHP script needs to be changed. I can visit the site perfectly in Lynx, Firefox and IE. Are you just trying to get the contents of the page, or is there something special you're doing? If it's just the contents you're after, try file_get_contents() if allow_url_fopen is set to TRUE for your PHP installation. http://php.net/file_get_contents http://php.net/allow_url_fopen Hope this helps, -- // Todd
Re: [PHP] CURL vs stream_socket_client
Hey there, First off I'd like to thank the Magento module's author for the support. The canada post shipping module now works! The problem was a little bit different then I saw in other posts and forums. Basically, the curl function wouldn't connect to the sellonline.canadapost.ca system on the desired port (3). In the curl section of code, we set opts.. and in the opts... we set the URL and the port #. For some reason this would not work. curl_setopt($ch, CURLOPT_URL, $url); curl_setopt($ch, CURLOPT_PORT, $port); So what ended up working was to add the port directly to the url and comment out the port option.. // basically this is what is being passed to the URL opt $url = 'http://sellonline.canadapost.ca:3'; curl_setopt($ch, CURLOPT_URL, $url); // and comment out the port opt //curl_setopt($ch, CURLOPT_PORT, $port); So, after some diggin' around on curl and great support by module's author problem solved. Canada Post shipping module now works. :) Yves Arsenault Love is the only force capable of transforming an enemy into a friend. --Martin Luther King, Jr. On Fri, Jan 30, 2009 at 4:18 PM, Yves Arsenault yves.arsena...@gmail.comwrote: Hi all, It really seems to me that somehow when the curl command is run that it's not posting the request to port 3 to sellonline.canadapost.ca. That's the impression I'm under. Any ideas? Yves Arsenault Love is the only force capable of transforming an enemy into a friend. --Martin Luther King, Jr. On Fri, Jan 30, 2009 at 12:40 PM, Yves Arsenault yves.arsena...@gmail.com wrote: Hi Todd, First off.. thanks for the reply. I've dumped the xml data and tried to view it in Firefox... commented out the curl code (and all the other stuff) just to see what exactly was getting passed. Firefox did not display it as it does xml... like when I got directly to that xml file with firefox, I get the This XML file does not appear to have any style information associated with it. The document tree is shown below. (grey bar)... it always shows properly formed xml documents like that for me (collapsible)... But, when I dump it in the PHP script using: print $eparcelRequest; It will just display the values... don't see any collapsible xml tags like when I view the xml directly. So I was thinking it might be in the format sent.. I'll try to set the user agent. Thanks! Yves Arsenault Love is the only force capable of transforming an enemy into a friend. --Martin Luther King, Jr. On Fri, Jan 30, 2009 at 12:19 PM, Boyd, Todd M. tmbo...@ccis.edu wrote: -Original Message- From: Yves Arsenault [mailto:yves.arsena...@gmail.com] Sent: Friday, January 30, 2009 9:40 AM To: PHP List Subject: [PHP] CURL vs stream_socket_client Hi there, I'm not in PHP on a daily basis, but from time to time I get to program a little in PHP for certain projects. Recently, I've installed a Magento system... and I've installed a canada post shipping module for this system. One thing that seems to be tripping it up is that it's using curl. curl is attempting to send xml data to canada post's server on port 3. The response from canada post's system is always an html page... I've run a test using stream_socket_client and have gotten it to work, the canada post's system responds with the corresponding XML data with shipping info. I'm wondering if there are any curlopt_ options that should be set that aren't currently set... Could be that curl is not sending properly formatted XML. This may be a stretch, but try to set the User Agent string to mimic Firefox before sending your request. I wrote a PHP scrape to gather guild member information from the World Of Warcraft Armory [1] a few years back, and it kept giving me HTML instead of XML until I used the User Agent of a browser that most sites accept is capable of parsing XML. A lot of sites conditionally send IE (and unknown browsers) XHTML instead of XML if there is any transformation of data involved. Also--your comment Could be that curl is not sending properly formatted XML is possible. Have you looked at the resulting HTML that is sent to curl to make sure that it's not a server error message? If it's the results you were looking for (but in XHTML format instead of XML), I'll bet it's a User Agent thing. Just a thought... // Todd
RE: [PHP] CURL vs stream_socket_client
-Original Message- From: Yves Arsenault [mailto:yves.arsena...@gmail.com] Sent: Friday, January 30, 2009 9:40 AM To: PHP List Subject: [PHP] CURL vs stream_socket_client Hi there, I'm not in PHP on a daily basis, but from time to time I get to program a little in PHP for certain projects. Recently, I've installed a Magento system... and I've installed a canada post shipping module for this system. One thing that seems to be tripping it up is that it's using curl. curl is attempting to send xml data to canada post's server on port 3. The response from canada post's system is always an html page... I've run a test using stream_socket_client and have gotten it to work, the canada post's system responds with the corresponding XML data with shipping info. I'm wondering if there are any curlopt_ options that should be set that aren't currently set... Could be that curl is not sending properly formatted XML. This may be a stretch, but try to set the User Agent string to mimic Firefox before sending your request. I wrote a PHP scrape to gather guild member information from the World Of Warcraft Armory [1] a few years back, and it kept giving me HTML instead of XML until I used the User Agent of a browser that most sites accept is capable of parsing XML. A lot of sites conditionally send IE (and unknown browsers) XHTML instead of XML if there is any transformation of data involved. Also--your comment Could be that curl is not sending properly formatted XML is possible. Have you looked at the resulting HTML that is sent to curl to make sure that it's not a server error message? If it's the results you were looking for (but in XHTML format instead of XML), I'll bet it's a User Agent thing. Just a thought... // Todd -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] CURL vs stream_socket_client
Hi Todd, First off.. thanks for the reply. I've dumped the xml data and tried to view it in Firefox... commented out the curl code (and all the other stuff) just to see what exactly was getting passed. Firefox did not display it as it does xml... like when I got directly to that xml file with firefox, I get the This XML file does not appear to have any style information associated with it. The document tree is shown below. (grey bar)... it always shows properly formed xml documents like that for me (collapsible)... But, when I dump it in the PHP script using: print $eparcelRequest; It will just display the values... don't see any collapsible xml tags like when I view the xml directly. So I was thinking it might be in the format sent.. I'll try to set the user agent. Thanks! Yves Arsenault Love is the only force capable of transforming an enemy into a friend. --Martin Luther King, Jr. On Fri, Jan 30, 2009 at 12:19 PM, Boyd, Todd M. tmbo...@ccis.edu wrote: -Original Message- From: Yves Arsenault [mailto:yves.arsena...@gmail.com] Sent: Friday, January 30, 2009 9:40 AM To: PHP List Subject: [PHP] CURL vs stream_socket_client Hi there, I'm not in PHP on a daily basis, but from time to time I get to program a little in PHP for certain projects. Recently, I've installed a Magento system... and I've installed a canada post shipping module for this system. One thing that seems to be tripping it up is that it's using curl. curl is attempting to send xml data to canada post's server on port 3. The response from canada post's system is always an html page... I've run a test using stream_socket_client and have gotten it to work, the canada post's system responds with the corresponding XML data with shipping info. I'm wondering if there are any curlopt_ options that should be set that aren't currently set... Could be that curl is not sending properly formatted XML. This may be a stretch, but try to set the User Agent string to mimic Firefox before sending your request. I wrote a PHP scrape to gather guild member information from the World Of Warcraft Armory [1] a few years back, and it kept giving me HTML instead of XML until I used the User Agent of a browser that most sites accept is capable of parsing XML. A lot of sites conditionally send IE (and unknown browsers) XHTML instead of XML if there is any transformation of data involved. Also--your comment Could be that curl is not sending properly formatted XML is possible. Have you looked at the resulting HTML that is sent to curl to make sure that it's not a server error message? If it's the results you were looking for (but in XHTML format instead of XML), I'll bet it's a User Agent thing. Just a thought... // Todd
Re: [PHP] CURL vs stream_socket_client
Hi all, It really seems to me that somehow when the curl command is run that it's not posting the request to port 3 to sellonline.canadapost.ca. That's the impression I'm under. Any ideas? Yves Arsenault Love is the only force capable of transforming an enemy into a friend. --Martin Luther King, Jr. On Fri, Jan 30, 2009 at 12:40 PM, Yves Arsenault yves.arsena...@gmail.comwrote: Hi Todd, First off.. thanks for the reply. I've dumped the xml data and tried to view it in Firefox... commented out the curl code (and all the other stuff) just to see what exactly was getting passed. Firefox did not display it as it does xml... like when I got directly to that xml file with firefox, I get the This XML file does not appear to have any style information associated with it. The document tree is shown below. (grey bar)... it always shows properly formed xml documents like that for me (collapsible)... But, when I dump it in the PHP script using: print $eparcelRequest; It will just display the values... don't see any collapsible xml tags like when I view the xml directly. So I was thinking it might be in the format sent.. I'll try to set the user agent. Thanks! Yves Arsenault Love is the only force capable of transforming an enemy into a friend. --Martin Luther King, Jr. On Fri, Jan 30, 2009 at 12:19 PM, Boyd, Todd M. tmbo...@ccis.edu wrote: -Original Message- From: Yves Arsenault [mailto:yves.arsena...@gmail.com] Sent: Friday, January 30, 2009 9:40 AM To: PHP List Subject: [PHP] CURL vs stream_socket_client Hi there, I'm not in PHP on a daily basis, but from time to time I get to program a little in PHP for certain projects. Recently, I've installed a Magento system... and I've installed a canada post shipping module for this system. One thing that seems to be tripping it up is that it's using curl. curl is attempting to send xml data to canada post's server on port 3. The response from canada post's system is always an html page... I've run a test using stream_socket_client and have gotten it to work, the canada post's system responds with the corresponding XML data with shipping info. I'm wondering if there are any curlopt_ options that should be set that aren't currently set... Could be that curl is not sending properly formatted XML. This may be a stretch, but try to set the User Agent string to mimic Firefox before sending your request. I wrote a PHP scrape to gather guild member information from the World Of Warcraft Armory [1] a few years back, and it kept giving me HTML instead of XML until I used the User Agent of a browser that most sites accept is capable of parsing XML. A lot of sites conditionally send IE (and unknown browsers) XHTML instead of XML if there is any transformation of data involved. Also--your comment Could be that curl is not sending properly formatted XML is possible. Have you looked at the resulting HTML that is sent to curl to make sure that it's not a server error message? If it's the results you were looking for (but in XHTML format instead of XML), I'll bet it's a User Agent thing. Just a thought... // Todd
Re: [PHP] curl can't connect to my home server
On Tue, Jan 20, 2009 at 1:33 PM, Rene Veerman rene7...@gmail.com wrote: Hi, I've been stuck on this problem i'm having after i re-installed my linux (debian) machine last month.. I'm building a CMS with video import capabilities, but since it runs on shared hosting i need to make a cURL POST call to a URL on my home machine, which does video-conversion. This curl call has stopped working, curl_error() claiming couldn't connect to host. As a test, i'm simply calling up a url on my homeserver through curl.. The url is http://82.170.249.144/mediaBeez/sn.php The curl script is at http://veerman.ws/servagetest.php Any light you can shed on this is very greatly appreciated. I'm kinda stuck at a huge brick wall here.. I just tried from the command line and it's timing out. Check your os firewall/router firewall? -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] curl can't connect to my home server
On Jan 20, 2009, at 1:33 PM, Rene Veerman wrote: Hi, I've been stuck on this problem i'm having after i re-installed my linux (debian) machine last month.. I'm building a CMS with video import capabilities, but since it runs on shared hosting i need to make a cURL POST call to a URL on my home machine, which does video-conversion. This curl call has stopped working, curl_error() claiming couldn't connect to host. As a test, i'm simply calling up a url on my homeserver through curl.. The url is http://82.170.249.144/mediaBeez/sn.php The curl script is at http://veerman.ws/servagetest.php Any light you can shed on this is very greatly appreciated. I'm kinda stuck at a huge brick wall here.. This might seem a little simplistic but... You say it's your home server... Do you either have a static IP or know for sure that that is your current IP? -- Jason Pruim japr...@raoset.com 616.399.2355
Re: [PHP] Curl with asp pages....
On Wed, Dec 17, 2008 at 6:28 AM, ioannes ioan...@btinternet.com wrote: shiplu wrote: On Sun, Dec 7, 2008 at 8:22 AM, ioannes ioan...@btinternet.com wrote: shiplu wrote: When you are dealing with curl, anything can be done as long as its a HTTP request.Its all about sending HTTP headers and content. To parse HTML content you can use HTML parser. Regular expression may not work each time. Pattern changes over time. Download Wireshark. Collect 2 sample request and response packet from there. Make a format and use it with CURL. Thats it. So Simple. You never gonna need to know who is generating the site, PHP or ASP.NET. I downloaded Wireshark onto Windows XP, got as far as Capture Options from Ethernet, Capture Filter is host IP address of target page, click Start, go to browser and access page, Stop Wireshark, Save captured file or Export as HTTP object which gives me the source of the page again. Is this what you mean? What do you mean by make a format - do you mean for instance parse the page with string finder functions etc. How is this helping over identifying the correct POST variables (using LiveHTTP etc) of the request and feeding into a curl function? What do you mean by 'make a format' versus 'pattern changes over time' - is format a Wireshark function, if so where do I find it. Thanks, John make a format is not like a button in wireshirk that has label make a format and it will do everything for you. You have to do it yourself. By wireshirk you'll see every type of headers and contents for almost every type of protocols. So you'll use this soft for analyzing the http conversation. Data will not only be in content but also in headers. so parse both if needed. then use the same data and make successive requests. If you are using regular expression it will fail to match if pattern changes. Your pattern '/input type=hidden name=__VIEWSTATE id=__VIEWSTATE value=([^]*?) \// will match input type=hidden name=__VIEWSTATE id=__VIEWSTATE value=ABC7D5ACSE / but wont match input type=hidden name=__VIEWSTATE id=__VIEWSTATE value=ABC7D5ACSE. Do you see the difference?? It wont mach input type=hidden id=__VIEWSTATE name=__VIEWSTATE value=ABC7D5ACSE / too. Because the attributes order is changed. Your regex will not work but their website will render very well. to overcome this, you have to use html/xml parser. So you can go to input element. then look for name attribute and if the name attribute is __VIEWSTATE then fetch the value attributes content. To make any input element name, value attribute must be present. So your code will match every time. It wont fail in 99.99% case. Hope that make sense Yes, thanks. What HTML parser do you suggest? John For php there is a dom extension. Documentation can be found in http://www.php.net/dom Thanks. -- A K M Mokaddim http://talk.cmyweb.net http://twitter.com/shiplu Stop Top Posting !! -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] Curl with asp pages....
Ashley Sheridan wrote: There is a slight difference in how ASP handles multiple form fields that share the same name (as well as SELECT MULTIPLE lists) such that ASP does not need (and should not use) square brackets in field names the way PHP does. Andrew I thought that square brackets were actually part of the HTML spec, not PHP, so the fact that ASP.Net cannot use them is just another example of M$ doing their own thing again... Why did you think that? I think you'll find square brackets are a PHPism and that PHP is actually a freakish oddball in not interpreting key=barkey=fookey=quux as $_GET['key'] = array( 'bar', 'foo', 'quux' ); Contrast: * PERL CGI module: http://docstore.mik.ua/orelly/linux/cgi/ch05_02.htm * Python urlparse library: http://docs.python.org/library/urlparse.html#urlparse.parse_qs * Ruby on Rails: http://dev.rubyonrails.org/ticket/6645 IIRC there's nothing in the URI, HTML, or CGI specs that enforces how these should be interpreted. -- Benjamin Hawkes-Lewis -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] Curl with asp pages....
On Sun, Dec 7, 2008 at 8:22 AM, ioannes [EMAIL PROTECTED] wrote: shiplu wrote: When you are dealing with curl, anything can be done as long as its a HTTP request.Its all about sending HTTP headers and content. To parse HTML content you can use HTML parser. Regular expression may not work each time. Pattern changes over time. Download Wireshark. Collect 2 sample request and response packet from there. Make a format and use it with CURL. Thats it. So Simple. You never gonna need to know who is generating the site, PHP or ASP.NET. I downloaded Wireshark onto Windows XP, got as far as Capture Options from Ethernet, Capture Filter is host IP address of target page, click Start, go to browser and access page, Stop Wireshark, Save captured file or Export as HTTP object which gives me the source of the page again. Is this what you mean? What do you mean by make a format - do you mean for instance parse the page with string finder functions etc. How is this helping over identifying the correct POST variables (using LiveHTTP etc) of the request and feeding into a curl function? What do you mean by 'make a format' versus 'pattern changes over time' - is format a Wireshark function, if so where do I find it. Thanks, John make a format is not like a button in wireshirk that has label make a format and it will do everything for you. You have to do it yourself. By wireshirk you'll see every type of headers and contents for almost every type of protocols. So you'll use this soft for analyzing the http conversation. Data will not only be in content but also in headers. so parse both if needed. then use the same data and make successive requests. If you are using regular expression it will fail to match if pattern changes. Your pattern '/input type=hidden name=__VIEWSTATE id=__VIEWSTATE value=([^]*?) \// will match input type=hidden name=__VIEWSTATE id=__VIEWSTATE value=ABC7D5ACSE / but wont match input type=hidden name=__VIEWSTATE id=__VIEWSTATE value=ABC7D5ACSE. Do you see the difference?? It wont mach input type=hidden id=__VIEWSTATE name=__VIEWSTATE value=ABC7D5ACSE / too. Because the attributes order is changed. Your regex will not work but their website will render very well. to overcome this, you have to use html/xml parser. So you can go to input element. then look for name attribute and if the name attribute is __VIEWSTATE then fetch the value attributes content. To make any input element name, value attribute must be present. So your code will match every time. It wont fail in 99.99% case. Hope that make sense -- A K M Mokaddim http://talk.cmyweb.net http://twitter.com/shiplu Stop Top Posting !! -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] Curl with asp pages....
On Fri, 2008-11-28 at 01:13 -0500, Andrew Ballard wrote: On Thu, Nov 27, 2008 at 7:12 PM, ioannes [EMAIL PROTECTED] wrote: What are the differences between asp and non-asp pages when you are curling them? Apart from ,as referred to in php.net, you need to urlencode the post values... Do you also need to urlencode the variable names? And if the submit button on the page has javascript:WebForm_PostBackOptions, is that going to cause any complexity? I have a working php curling script which works on many sites, but this asp page returns error from the server (Error Tracking Code: from server). I have seen problems described on the internet with asp but apart from urlencode tip and that you need to submit the hidden VIEWSTATE etc variables, I have not seen any other tips. I previously got just the input page, so getting the error is actually progress, which I got once I urlencoded the inputs, but clearly the inputs don't make sense to the script somehow and suspect there is some general differences between asp and non-asp pages for this purpose. John It sounds like you're talking about ASP.NET rather than just ASP. (ASP is simple enough that CURL usually shouldn't have any problems.) ASP.NET added VIEWSTATE, which is a hidden form field that contains an encoded value representing the entire form. It's a way for the server to determine that the data posted matches the form that requested it, as well as handling server-side actions that are triggered by javascript that ASP.NET embeds in the form. It is very likely that you will need to read the correct value of this field from the site prior to sending your data. There is a slight difference in how ASP handles multiple form fields that share the same name (as well as SELECT MULTIPLE lists) such that ASP does not need (and should not use) square brackets in field names the way PHP does. Andrew I thought that square brackets were actually part of the HTML spec, not PHP, so the fact that ASP.Net cannot use them is just another example of M$ doing their own thing again... Ash www.ashleysheridan.co.uk -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] Curl with asp pages....
The one thing that's always tripped me up with ASP sites is that you have to add EVERY input, even the type=submit with the correct value from the one and only submit button on the page. Not sure what the ASP code monkeys are doing with their point-and-click UI, but I presume it's just a boilerplate operation that needs the submit input just in case they add a second button some day. The variable names shouldn't need URLencoding in any proper webapp, but try it and see. So check the form (and any JS) carefully to be sure you are not missing some INPUT element in your POST data. Use Firefox LiveHTTPHeaders to see what goes back-n-forth on a real working exchange as well. ASP developers tend to do a LOT of needless header(Location: ) bouncing with added/changed cookies as well, so you have to track all of that. -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] Curl with asp pages....
On Fri, Nov 28, 2008 at 9:59 AM, [EMAIL PROTECTED] wrote: The one thing that's always tripped me up with ASP sites is that you have to add EVERY input, even the type=submit with the correct value from the one and only submit button on the page. Again, I think that's the ASP.NET part. Not sure what the ASP code monkeys are doing with their point-and-click UI, but I presume it's just a boilerplate operation that needs the submit input just in case they add a second button some day. I've had the same problem with Zend_Form, honestly. The variable names shouldn't need URLencoding in any proper webapp, but try it and see. Well, they either need to contain valid URL characters or they need to be encoded. If you have a variable name that contains an equal sign (crazy, but can be technically valid since names are just strings -- to me it seems little less crazy than including square brackets in a variable name in HTML) it will have to be escaped. Use Firefox LiveHTTPHeaders to see what goes back-n-forth on a real working exchange as well. I agree. It's definitely a lifesaver. FWIW, there is a similar tool for IE too, if you ever find yourself stuck for one of those lovely cases where the site works fine in Firefox but not in IE. Andrew -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] Curl with asp pages....
On Fri, Nov 28, 2008 at 10:08 AM, [EMAIL PROTECTED] wrote: There is a slight difference in how ASP handles multiple form fields that share the same name (as well as SELECT MULTIPLE lists) such that ASP does not need (and should not use) square brackets in field names the way PHP does. Alas, not only doesn't it need them, it CANNOT use them, so you are stuck if you want to do something such as: name=foo[1] value=17 name=foo[5] value=47 name=foo[17] value=32 I was quite disappointed when I found out that ASP simply could not maintain a simple key-value pairing without jumping through hoops... Having worked in PHP long enough to realize the value of what you are talking about, I can see a use for it. It still seems odd to me, though. :-) It made for a whole 'nother verse in my Microsoft Sucks song strewn throughout the code comments of that ASP project. LOL Another verse was Boolean expressions don't short-circuit :-) Yeah, that got me for a while when I first started working in ASP after PHP and JavaScript. What a pain. Ah, and Can't moveFirst in an empty record set was another. Well, I got used to checking for EOF and/or BOF before trying to move around. Once I got used to it, I didn't find it that big of a deal. For those accustomed to PHP, there are a couple methods on the recordset object that will make things more familiar. One implodes the recordset with cell and row delimiters that are useful for packing the data into a CSV (providing that a value doesn't have commas or double quotes that need escaped); the other turns the recordset into a multi-dimensional array. As I recall, though, the indexes are reversed. Sadly, the song is lost as I failed to make a copy of that codebase before I moved on to another task. Sadly? Are you really pining for it? :-) Andrew -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] Curl with asp pages....
Andrew Ballard wrote:
On Fri, Nov 28, 2008 at 9:59 AM, [EMAIL PROTECTED] wrote:
The one thing that's always tripped me up with ASP sites is that you have to add EVERY
input, even the type=submit with the correct value from the one and only
submit button on the page.
Again, I think that's the ASP.NET part.
Not sure what the ASP code monkeys are doing with their point-and-click UI, but
I presume it's just a boilerplate operation that needs the submit input just in
case they add a second button some day.
I've had the same problem with Zend_Form, honestly.
The variable names shouldn't need URLencoding in any proper webapp, but try it
and see.
Well, they either need to contain valid URL characters or they need to
be encoded. If you have a variable name that contains an equal sign
(crazy, but can be technically valid since names are just strings --
to me it seems little less crazy than including square brackets in a
variable name in HTML) it will have to be escaped.
Use Firefox LiveHTTPHeaders to see what goes back-n-forth on a real working
exchange as well.
I agree. It's definitely a lifesaver. FWIW, there is a similar tool
for IE too, if you ever find yourself stuck for one of those lovely
cases where the site works fine in Firefox but not in IE.
Andrew
I agree too. Thank you, those responses were very useful, in particular
Firefox LiveHTTPHeaders. This gives the full post variables on the
submission of the form, using this I could get a correct response and
work back, compare the syntax and very carefully substitute my own
variables. I found that the code I was using to get the VIEWSTATE was
in fact correct:
preg_match('/input type=hidden name=__VIEWSTATE
id=__VIEWSTATE value=([^]*?) \//', $html, $matches); //- from
php.net curl php with slight amendment
$viewstate = $matches[1];
$viewstate = urlencode($viewstate);
but Firefox LiveHTTPHeaders then helped enormously in putting all the
other variables in the right place and avoid errors. I also found out
that the server was doing a logic check on one of the variables
(December was returned as 11 for the javascript calendar) which became
obvious once everything else was working.
John
--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] Curl with asp pages....
When you are dealing with curl, anything can be done as long as its a HTTP request.Its all about sending HTTP headers and content. To parse HTML content you can use HTML parser. Regular expression may not work each time. Pattern changes over time. Download Wireshark. Collect 2 sample request and response packet from there. Make a format and use it with CURL. Thats it. So Simple. You never gonna need to know who is generating the site, PHP or ASP.NET. -- A K M Mokaddim http://talk.cmyweb.net http://twitter.com/shiplu Stop Top Posting !!
Re: [PHP] Curl with asp pages....
On Thu, Nov 27, 2008 at 7:12 PM, ioannes [EMAIL PROTECTED] wrote: What are the differences between asp and non-asp pages when you are curling them? Apart from ,as referred to in php.net, you need to urlencode the post values... Do you also need to urlencode the variable names? And if the submit button on the page has javascript:WebForm_PostBackOptions, is that going to cause any complexity? I have a working php curling script which works on many sites, but this asp page returns error from the server (Error Tracking Code: from server). I have seen problems described on the internet with asp but apart from urlencode tip and that you need to submit the hidden VIEWSTATE etc variables, I have not seen any other tips. I previously got just the input page, so getting the error is actually progress, which I got once I urlencoded the inputs, but clearly the inputs don't make sense to the script somehow and suspect there is some general differences between asp and non-asp pages for this purpose. John It sounds like you're talking about ASP.NET rather than just ASP. (ASP is simple enough that CURL usually shouldn't have any problems.) ASP.NET added VIEWSTATE, which is a hidden form field that contains an encoded value representing the entire form. It's a way for the server to determine that the data posted matches the form that requested it, as well as handling server-side actions that are triggered by javascript that ASP.NET embeds in the form. It is very likely that you will need to read the correct value of this field from the site prior to sending your data. There is a slight difference in how ASP handles multiple form fields that share the same name (as well as SELECT MULTIPLE lists) such that ASP does not need (and should not use) square brackets in field names the way PHP does. Andrew -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] CURL de-bugging: So why am I not getting the results page on the target site?
I would like someone to help me on this outside the group discussion. Is there anyone out there that would look at a curl problem with me? Thanks, John ioannes wrote: My current theory on this is that the initial input page creates a per-session cookie. Is CURL able to send this when the page is submitted and if so how do I find out the name and value of the cookie as per my reading it is not stored on the computer, though I reckon it must be there somewhere. The initial page is in this format: https://www.targetsite.com/subdir/page.asp?serialno=GM201OtherCode=NextCode= So I have various GET parameters. The initial input page also has POST parameters which I can see the in page source, and so send via CURL. I cannot find any javascript doing a set_cookie, including in any included files of which there are none anyway. But there is some session going on as the serialno parameter is not in a hidden field or explicitly set cookie, so it must be a per-session cookie maybe with a different name, I reason. How do I find out about these? John ioannes wrote: My code is as below. It comes back with 'Bad session variable name - CompanySerialNo' from the site.but the COOKIEJAR does not show this variable name and it is not sent, it just shows: www.targetsite.comFALSE/FALSE0 ASPSESSIONIDQCSQDTABLKAONANAFJPNMFFECLFNCLBP There is a serialno but that is sent in the GET (URL below). Question is: What to test now? I am trying to get a results page from an input page. What code below is trying to do is access the page, get any cookies set then try the page again with the relevant inputs. ? $url=https://www.targetsite.com/subdir/page.asp?serialno=GM201OtherCode=NextCode=;; $ch = curl_init(); curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE); curl_setopt ($ch, CURLOPT_COOKIEJAR, cookies.txt); curl_setopt ($ch, CURLOPT_COOKIEFILE, cookies.txt); curl_setopt($ch, CURLOPT_HEADER, 0); curl_setopt($ch, CURLOPT_AUTOREFERER, true); curl_setopt($ch, CURLOPT_URL, $url); /* //GET list from submitting POST form as GET https://www.shortstay-london.com/checkavail.asp? 1 - clock=+09%3A54 2 - StartDay=6 3 - StartMonth=September+%3A+2008 4 - EndDay=13 5 - EndMonth=September+%3A+2008 13 - CheckThis=Check+This use this list to create POST data */ curl_setopt($ch, CURLOPT_POST, 1); $curlPost=array(); $curlPost[clock]= 09:54; $curlPost[StartDay]=6; $curlPost[StartMonth]=September : 2008; $curlPost[EndDay]=13; $curlPost[EndMonth]=September : 2008; //etc $curlPost[CheckThis]=Check This; curl_setopt($ch, CURLOPT_POSTFIELDS,$curlPost); curl_setopt($ch, CURLOPT_FOLLOWLOCATION ,1); curl_setopt ($ch, CURLOPT_RETURNTRANSFER, 1); $store = curl_exec ($ch); curl_close ($ch); print($store); ? -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] CURL de-bugging: So why am I not getting the results page on the target site?
My current theory on this is that the initial input page creates a per-session cookie. Is CURL able to send this when the page is submitted and if so how do I find out the name and value of the cookie as per my reading it is not stored on the computer, though I reckon it must be there somewhere. The initial page is in this format: https://www.targetsite.com/subdir/page.asp?serialno=GM201OtherCode=NextCode= So I have various GET parameters. The initial input page also has POST parameters which I can see the in page source, and so send via CURL. I cannot find any javascript doing a set_cookie, including in any included files of which there are none anyway. But there is some session going on as the serialno parameter is not in a hidden field or explicitly set cookie, so it must be a per-session cookie maybe with a different name, I reason. How do I find out about these? John ioannes wrote: My code is as below. It comes back with 'Bad session variable name - CompanySerialNo' from the site.but the COOKIEJAR does not show this variable name and it is not sent, it just shows: www.targetsite.comFALSE/FALSE0 ASPSESSIONIDQCSQDTABLKAONANAFJPNMFFECLFNCLBP There is a serialno but that is sent in the GET (URL below). Question is: What to test now? I am trying to get a results page from an input page. What code below is trying to do is access the page, get any cookies set then try the page again with the relevant inputs. ? $url=https://www.targetsite.com/subdir/page.asp?serialno=GM201OtherCode=NextCode=;; $ch = curl_init(); curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE); curl_setopt ($ch, CURLOPT_COOKIEJAR, cookies.txt); curl_setopt ($ch, CURLOPT_COOKIEFILE, cookies.txt); curl_setopt($ch, CURLOPT_HEADER, 0); curl_setopt($ch, CURLOPT_AUTOREFERER, true); curl_setopt($ch, CURLOPT_URL, $url); /* //GET list from submitting POST form as GET https://www.shortstay-london.com/checkavail.asp? 1 - clock=+09%3A54 2 - StartDay=6 3 - StartMonth=September+%3A+2008 4 - EndDay=13 5 - EndMonth=September+%3A+2008 13 - CheckThis=Check+This use this list to create POST data */ curl_setopt($ch, CURLOPT_POST, 1); $curlPost=array(); $curlPost[clock]= 09:54; $curlPost[StartDay]=6; $curlPost[StartMonth]=September : 2008; $curlPost[EndDay]=13; $curlPost[EndMonth]=September : 2008; //etc $curlPost[CheckThis]=Check This; curl_setopt($ch, CURLOPT_POSTFIELDS,$curlPost); curl_setopt($ch, CURLOPT_FOLLOWLOCATION ,1); curl_setopt ($ch, CURLOPT_RETURNTRANSFER, 1); $store = curl_exec ($ch); curl_close ($ch); print($store); ? -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] CURL de-bugging: So why am I not getting the results page on the target site?
The way this page works is you access index.asp?SerialNo=abc123 from a link, that is re-directed to a frameset containing the main page, main.asp, just that. main.asp does not have any input fields showing the SerialNo, there is a comment with it but that is all. Fom this page, you input some dates and submit, and the form action is on a page like /checkthis.asp. What I can't see is how /checkthis.asp knows the SerialNo, which it needs. Is there some other way that asp pages transfer information between pages? I looked at the cache for IE after each page load and it does not have SerialNo as cookies or any cookies from this site other than google conversion tracking. On main.asp the button that submits the form does not refer to the SerialNo either. Nevertheless the results page creates the relevant data from the database. But it does produce error messages like, The Object has been Moved here, with my site plus ?lngSessionId=234984987SerialNo=abd123WinMode=AgentCode=CustCode=. If I add these parameters to the target URL in the CURL script it looks for displaymode (the message with my url in the display line at end of browser window: www.mysite.com/displaymode=). John ioannes wrote: My current theory on this is that the initial input page creates a per-session cookie. Is CURL able to send this when the page is submitted and if so how do I find out the name and value of the cookie as per my reading it is not stored on the computer, though I reckon it must be there somewhere. The initial page is in this format: https://www.targetsite.com/subdir/page.asp?serialno=GM201OtherCode=NextCode= So I have various GET parameters. The initial input page also has POST parameters which I can see the in page source, and so send via CURL. I cannot find any javascript doing a set_cookie, including in any included files of which there are none anyway. But there is some session going on as the serialno parameter is not in a hidden field or explicitly set cookie, so it must be a per-session cookie maybe with a different name, I reason. How do I find out about these? John ioannes wrote: My code is as below. It comes back with 'Bad session variable name - CompanySerialNo' from the site.but the COOKIEJAR does not show this variable name and it is not sent, it just shows: www.targetsite.comFALSE/FALSE0 ASPSESSIONIDQCSQDTABLKAONANAFJPNMFFECLFNCLBP There is a serialno but that is sent in the GET (URL below). Question is: What to test now? I am trying to get a results page from an input page. What code below is trying to do is access the page, get any cookies set then try the page again with the relevant inputs. ? $url=https://www.targetsite.com/subdir/page.asp?serialno=GM201OtherCode=NextCode=;; $ch = curl_init(); curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE); curl_setopt ($ch, CURLOPT_COOKIEJAR, cookies.txt); curl_setopt ($ch, CURLOPT_COOKIEFILE, cookies.txt); curl_setopt($ch, CURLOPT_HEADER, 0); curl_setopt($ch, CURLOPT_AUTOREFERER, true); curl_setopt($ch, CURLOPT_URL, $url); /* //GET list from submitting POST form as GET https://www.shortstay-london.com/checkavail.asp? 1 - clock=+09%3A54 2 - StartDay=6 3 - StartMonth=September+%3A+2008 4 - EndDay=13 5 - EndMonth=September+%3A+2008 13 - CheckThis=Check+This use this list to create POST data */ curl_setopt($ch, CURLOPT_POST, 1); $curlPost=array(); $curlPost[clock]= 09:54; $curlPost[StartDay]=6; $curlPost[StartMonth]=September : 2008; $curlPost[EndDay]=13; $curlPost[EndMonth]=September : 2008; //etc $curlPost[CheckThis]=Check This; curl_setopt($ch, CURLOPT_POSTFIELDS,$curlPost); curl_setopt($ch, CURLOPT_FOLLOWLOCATION ,1); curl_setopt ($ch, CURLOPT_RETURNTRANSFER, 1); $store = curl_exec ($ch); curl_close ($ch); print($store); ? -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] CURL de-bugging: So why am I not getting the results page on the target site?
My code is as below. It comes back with 'Bad session variable name - CompanySerialNo' from the site.but the COOKIEJAR does not show this variable name and it is not sent, it just shows: www.targetsite.comFALSE/FALSE0 ASPSESSIONIDQCSQDTABLKAONANAFJPNMFFECLFNCLBP There is a serialno but that is sent in the (URL below). Question is: What to test now? I am trying to get a results page from an input page. What code below is trying to do is access the page, get any cookies set then try the page again with the relevant inputs. ? $url=https://www.targetsite.com/subdir/page.asp?serialno=GM201OtherCode=NextCode=;; $ch = curl_init(); curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE); curl_setopt ($ch, CURLOPT_COOKIEJAR, cookies.txt); curl_setopt ($ch, CURLOPT_COOKIEFILE, cookies.txt); curl_setopt($ch, CURLOPT_HEADER, 0); curl_setopt($ch, CURLOPT_AUTOREFERER, true); curl_setopt($ch, CURLOPT_URL, $url); /* //GET list from submitting POST form as GET https://www.shortstay-london.com/checkavail.asp? 1 - clock=+09%3A54 2 - StartDay=6 3 - StartMonth=September+%3A+2008 4 - EndDay=13 5 - EndMonth=September+%3A+2008 13 - CheckThis=Check+This use this list to create POST data */ curl_setopt($ch, CURLOPT_POST, 1); $curlPost=array(); $curlPost[clock]= 09:54; $curlPost[StartDay]=6; $curlPost[StartMonth]=September : 2008; $curlPost[EndDay]=13; $curlPost[EndMonth]=September : 2008; //etc $curlPost[CheckThis]=Check This; curl_setopt($ch, CURLOPT_POSTFIELDS,$curlPost); curl_setopt($ch, CURLOPT_FOLLOWLOCATION ,1); curl_setopt ($ch, CURLOPT_RETURNTRANSFER, 1); $store = curl_exec ($ch); curl_close ($ch); print($store); ? -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] CURL de-bugging: So why am I not getting the results page on the target site?
Thanks Chris and Andrew, An interesting article here on VIEWSTATE in asp: http://www.dotnetjohn.com/articles.aspx?articleid=71 refers to MAC encoding using SHA1 or MD5, alternatively Triple DES symmetric algorithm. However, in either event, VIEWSTATE seems to be just what is sent by the server which the server expects to receive back unmodified. I tried the same input form as on the target server on a test page on my site with the action on the form to the target server and that works OK to get the results page on the target server. I have also tested on my server that the CURL POST variables are giving sensible inputs like the ones that the page would produce on GET. In the process, I also learned that colons in the POST variable names don't need to be changed for HTML encoding, spaces in the variable values do need to be, anyway the POST variables seem to work OK as I said. Whatever is or is not being sent by the script as opposed to the input form on a test page of my site is the difference that I am looking for. I will try another site see can I learn anything in the process. Oh, and decoding the string as suggested (print_r(base64_decode($view_state_string));) gives one value in the array and a mixture of English and other characters plus most of the page: eg d 2Ö8 Version=0.0.0.0, and other characters that do not copy to this email etc. As above, I don't see that this is interfering with CURLing the results page if it is simply sent back in the post. John -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
RE: [PHP] CURL de-bugging: So why am I not getting the results page on the target site?
-Original Message- From: Chris [mailto:[EMAIL PROTECTED] Sent: Tuesday, July 01, 2008 9:42 PM To: [EMAIL PROTECTED] Cc: php-general@lists.php.net Subject: Re: [PHP] CURL de-bugging: So why am I not getting the results page on the target site? ioannes wrote: I didn't get any brave response on this, but given the other thread on 'encription' I was wondering could anyone decrypt the __VIEWSTATE string at the end of this message. It is part of the input page whose results page I am trying to retrieve back onto my server for further php work. I replicated the source from that input page onto a page on my server, and when I click the submit button it correctly goes to the target results page, on the other site though, however it did not work without the whole of the string below. The experiment proved though that without the __VIEWSTATE the results page will not return. So I am just wondering, as I have not been able to repeat this using curl, what the is included in that string. There's a challenge for anyone with whatever resources it takes. echo base64_decode($view_state_string); viewstate in asp.net is like sessions in php (I believe, I could be completely wrong :P). Ehrm... you're not *completely* wrong. ;) I do most of my programming at work in ASP.NET/VB.NET, and I've come to understand that the VIEWSTATE in an ASP.NET page is more or less for retaining form values and client-side settings. IIS/ASP.NET still uses sessions--and in much the same way as PHP. Todd Boyd Web Programmer -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] CURL de-bugging: So why am I not getting the results page on the target site?
I didn't get any brave response on this, but given the other thread on 'encription' I was wondering could anyone decrypt the __VIEWSTATE string at the end of this message. It is part of the input page whose results page I am trying to retrieve back onto my server for further php work. I replicated the source from that input page onto a page on my server, and when I click the submit button it correctly goes to the target results page, on the other site though, however it did not work without the whole of the string below. The experiment proved though that without the __VIEWSTATE the results page will not return. So I am just wondering, as I have not been able to repeat this using curl, what the is included in that string. There's a challenge for anyone with whatever resources it takes. John ioannes wrote: For those that like CURL and calendars. ... VIEWSTATE curl_setopt($ch,
Re: [PHP] CURL de-bugging: So why am I not getting the results page on the target site?
ioannes wrote: I didn't get any brave response on this, but given the other thread on 'encription' I was wondering could anyone decrypt the __VIEWSTATE string at the end of this message. It is part of the input page whose results page I am trying to retrieve back onto my server for further php work. I replicated the source from that input page onto a page on my server, and when I click the submit button it correctly goes to the target results page, on the other site though, however it did not work without the whole of the string below. The experiment proved though that without the __VIEWSTATE the results page will not return. So I am just wondering, as I have not been able to repeat this using curl, what the is included in that string. There's a challenge for anyone with whatever resources it takes. echo base64_decode($view_state_string); viewstate in asp.net is like sessions in php (I believe, I could be completely wrong :P). -- Postgresql php tutorials http://www.designmagick.com/ -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] CURL de-bugging: So why am I not getting the results page on the target site?
Chris wrote: ioannes wrote: I didn't get any brave response on this, but given the other thread on 'encription' I was wondering could anyone decrypt the __VIEWSTATE string at the end of this message. It is part of the input page whose results page I am trying to retrieve back onto my server for further php work. I replicated the source from that input page onto a page on my server, and when I click the submit button it correctly goes to the target results page, on the other site though, however it did not work without the whole of the string below. The experiment proved though that without the __VIEWSTATE the results page will not return. So I am just wondering, as I have not been able to repeat this using curl, what the is included in that string. There's a challenge for anyone with whatever resources it takes. echo base64_decode($view_state_string); or maybe print_r(base64_decode($view_state_string)); I don't know if it will return a string or an array or something else. -- Postgresql php tutorials http://www.designmagick.com/ -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] CURL de-bugging: So why am I not getting the results page on the target site?
On Tue, Jul 1, 2008 at 5:23 PM, ioannes [EMAIL PROTECTED] wrote: I didn't get any brave response on this, but given the other thread on 'encription' I was wondering could anyone decrypt the __VIEWSTATE string at the end of this message. It is part of the input page whose results page I am trying to retrieve back onto my server for further php work. I replicated the source from that input page onto a page on my server, and when I click the submit button it correctly goes to the target results page, on the other site though, however it did not work without the whole of the string below. The experiment proved though that without the __VIEWSTATE the results page will not return. So I am just wondering, as I have not been able to repeat this using curl, what the is included in that string. There's a challenge for anyone with whatever resources it takes. John ioannes wrote: For those that like CURL and calendars. ... VIEWSTATE curl_setopt($ch,
Re: [PHP] cuRL script won't submit
I added more info below and corrected the spelling of a word - very sorry to add more to your already full mailbox why will this script not submit the form - it does input data into the form - but not submit it. below is the script and then the html form.. thanks #!/usr/bin/php ?php $ch = curl_init(); curl_setopt($ch, CURLOPT_RETURNTRANSFER,1); curl_setopt($ch, CURLOPT_COOKIEFILE, /temp/cookie.txt); curl_setopt($ch, CURLOPT_URL,http://192.168.10.10/?link=answerl=cqid=25AA98zRa;;); curl_setopt ($ch, CURLOPT_POSTFIELDS, link=answerqid=25AA98zRa.crumb=x88textarea=here+ya+goprev_ans_page=e=0submit1=Submit); $result = curl_exec ($ch); curl_close ($ch); echo $result; ? /// below is the HTML form \\\ form action=/question/act;_lt=Aktsgb5HMZ0vBEUSHj_3I1YdzKIX;_lv=3 onsubmit=this.submit1.disabled=true; method=post name=template_form input type=hidden value=answer name=link/ input type=hidden value=25AA98zRa name=qid/ input type=hidden value=x88 name=.crumb/ input type=hidden value= name=textarea/ input type=hidden value= name=refer/ input type=hidden value= name=prev_ans_page/ input type=hidden value=0 name=e/ input class=button type=submit name=submit1 value=Submit/ /form You rock. That's why Blockbuster's offering you one month of Blockbuster Total Access, No Cost. http://tc.deals.yahoo.com/tc/blockbuster/text5.com -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php You rock. That's why Blockbuster's offering you one month of Blockbuster Total Access, No Cost. http://tc.deals.yahoo.com/tc/blockbuster/text5.com
Re: [PHP] cuRL script won't submit
On Wed, Apr 2, 2008 at 9:24 AM, Test User [EMAIL PROTECTED] wrote: why will this script not submit the form - it does input data into the form - but not submit it. below is the script and then the html form.. thanks [snip!] curl_setopt($ch, CURLOPT_URL,http://192.168.10.10/?link=answerl=cqid=25AA98zRa;;); [snip!] /// below is the HTML form \\\ form action=/question/act;_lt=Aktsgb5HMZ0vBEUSHj_3I1YdzKIX;_lv=3 onsubmit=this.submit1.disabled=true; method=post name=template_form [snip!] Because you're submitting it to the wrong page. What you're doing will submit to the default (index) page in that directory, as opposed to the form action. Change this: curl_setopt($ch, CURLOPT_URL,http://192.168.10.10/?link=answerl=cqid=25AA98zRa;;); curl_setopt ($ch, CURLOPT_POSTFIELDS, link=answerqid=25AA98zRa.crumb=x88textarea=here+ya+goprev_ans_page=e=0submit1=Submit); To this: curl_setopt($ch, CURLOPT_URL,http://192.168.10.10/question/act;_lt=Aktsgb5HMZ0vBEUSHj_3I1YdzKIX;_lv=3;); curl_setopt ($ch, CURLOPT_POSTFIELDS, link=answerqid=25AA98zRa.crumb=x88textarea=here+ya+goprev_ans_page=e=0submit1=Submit); Also, note two things: 1.) I removed the semicolon from the end of the CURLOPT_URL value (between the quotes and parentheses). 2.) The form action looks like it's most likely dynamic, so you'll have to think of a workaround yourself if it is. -- /Daniel P. Brown Forensic Services, Senior Unix Engineer 1+ (570-) 362-0283 -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] cuRL script won't submit
many thanks.. I have been working on this for weeks! now I have to figure out how to extract that dynamic URL from the form before I post.. - Original Message From: Daniel Brown [EMAIL PROTECTED] To: Test User [EMAIL PROTECTED] Cc: php-general@lists.php.net Sent: Wednesday, April 2, 2008 8:54:16 AM Subject: Re: [PHP] cuRL script won't submit On Wed, Apr 2, 2008 at 9:24 AM, Test User [EMAIL PROTECTED] wrote: why will this script not submit the form - it does input data into the form - but not submit it. below is the script and then the html form.. thanks [snip!] curl_setopt($ch, CURLOPT_URL,http://192.168.10.10/?link=answerl=cqid=25AA98zRa;;); [snip!] /// below is the HTML form \\\ form action=/question/act;_lt=Aktsgb5HMZ0vBEUSHj_3I1YdzKIX;_lv=3 onsubmit=this.submit1.disabled=true; method=post name=template_form [snip!] Because you're submitting it to the wrong page. What you're doing will submit to the default (index) page in that directory, as opposed to the form action. Change this: curl_setopt($ch, CURLOPT_URL,http://192.168.10.10/?link=answerl=cqid=25AA98zRa;;); curl_setopt ($ch, CURLOPT_POSTFIELDS, link=answerqid=25AA98zRa.crumb=x88textarea=here+ya+goprev_ans_page=e=0submit1=Submit); To this: curl_setopt($ch, CURLOPT_URL,http://192.168.10.10/question/act;_lt=Aktsgb5HMZ0vBEUSHj_3I1YdzKIX;_lv=3;); curl_setopt ($ch, CURLOPT_POSTFIELDS, link=answerqid=25AA98zRa.crumb=x88textarea=here+ya+goprev_ans_page=e=0submit1=Submit); Also, note two things: 1.) I removed the semicolon from the end of the CURLOPT_URL value (between the quotes and parentheses). 2.) The form action looks like it's most likely dynamic, so you'll have to think of a workaround yourself if it is. -- /Daniel P. Brown Forensic Services, Senior Unix Engineer 1+ (570-) 362-0283 You rock. That's why Blockbuster's offering you one month of Blockbuster Total Access, No Cost. http://tc.deals.yahoo.com/tc/blockbuster/text5.com -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
