Re: Website scraping - How can I load a 'partial' page?

2017-12-13 Thread Mike Bonner via use-livecode
Hmm, or use range as mentioned in my other mail. If the server supports range requests you can set your headers to include-- Range: bytes=0-2000to get the first 2000 bytes. or use curl with -r 0-2000 but i have yet to find a page that will return only a range. Apparently you can find out if

Re: Website scraping - How can I load a 'partial' page?

2017-12-13 Thread Rick Harrison via use-livecode
Hi Roger, I don’t know who’s webpage is that you are scraping, but if it is a third party’s webpage make sure that you are not violating their terms of agreement or infringing on their copyright. You might want to ask for their permission to do so, to make sure you are safe and legal. If it is y

Re: Website scraping - How can I load a 'partial' page?

2017-12-13 Thread Mike Bonner via use-livecode
I suppose one could use sockets and partial GET requests (using a range: header), but i suspect it would be easier to just use an intermediary server to handle things. To test, I set up an extremely simple page with the following: http://url.goes.here put char 1 to 6000 of url tpage -- request

Website scraping - How can I load a 'partial' page?

2017-12-13 Thread Roger Eller via use-livecode
I have a webpage that I grab with LiveCode, then parse out what I need. The data I keep is within the first 1/4th of the page. Rather than loading the entire page into a variable or a browser object, how can I load just the portion that I need and then stop the transmission instead of wasting the