Maybe I should have mentioned that I use "wget" or "curl" to get a page.
As an example of how I might use J to extract information once I've done
this, say I've used "wget" to get the page at slashdot.org as the file
"eg.html". I would then break it up into lines like this:
$lns=. <;._2 ] LF (] , [ #~ [ ~: [: {: ]) CR-.~fread
'c:/amisc/J/NYCJUG/201509/eg.html'
4754
If I were interested in extracting information about the titles of articles
and their links on this page, I'd first identify a string that marks the
lines in which I'm interested and select those lines:
idstr=. '<span id="title'
>1242{lns
<span id="title-74449271"> <a onclick="return
toggle_fh_body_wrap_return(this);" href="//
tech.slashdot.org/story/15/07/14/1214207/65000-land-rovers-recalled-due-to-software-bug">65,000+
Land Rovers Recalled Due To Software Bug</a></span>
idstr E. >1242{lns
0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 ...
$sel=. lns#~+./&>(<idstr) E.&.>lns
15
Then I'd work on one of those lines to develop code to extract the parts in
which I'm interested:
ll=. >1242{lns
I. 'href="' E. ll
86
dropUpto=: ]}.~([:#[)+[:{.[:I.[E.]
'href="' dropUpto ll
//
tech.slashdot.org/story/15/07/14/1214207/65000-land-rovers-recalled-due-to-software-bug">65,000+
Land Rovers Recalled Due To Software Bug</a></span>
takeUntil=: ]{.~[:{.[:I.[E.]
'"' takeUntil 'href="' dropUpto ll
//
tech.slashdot.org/story/15/07/14/1214207/65000-land-rovers-recalled-due-to-software-bug
getLink=: '"' takeUntil 'href="' dropUpto ]
getTitle=: '</a>' takeUntil '">' dropUpto 'href="' dropUpto ]
Checking that this gives me what I want:
getTitle ll
65,000+ Land Rovers Recalled Due To Software Bug
Combining these:
extractLinksTitles=: [:(getTitle;getLink)&>]#~[:+./&>(<'<span
id="title') E.&.>]
And testing this:
$extractLinksTitles lns
15 2
2{.extractLinksTitles lns
+------------------------------------------------+------------------------------------------------------------------------------------------------+
|65,000+ Land Rovers Recalled Due To Software Bug|//
tech.slashdot.org/story/15/07/14/1214207/65000-land-rovers-recalled-due-to-software-bug
|
+------------------------------------------------+------------------------------------------------------------------------------------------------+
|For Microsoft, Windows 10 Charity Begins At Home|//
developers.slashdot.org/story/15/07/14/0436218/for-microsoft-windows-10-charity-begins-at-home|
+------------------------------------------------+------------------------------------------------------------------------------------------------+
There are undoubtedly more sophisticated and general ways to do this -
other languages have implemented full-blown HTML parsers and selectors -
but I've found this sort of quick-and-dirty approach sufficient for most
basic scraping.
On Tue, Jul 14, 2015 at 9:41 AM, chris burke <[email protected]> wrote:
> AFAIK there is nothing in J that would improve on the usual tools like wget
> or httrack.
>
> However, once the pages were downloaded, I might use J to fix them up, if
> needed.
>
> On 14 July 2015 at 05:19, Ryan Eckbo <[email protected]> wrote:
>
> > Hi everyone,
> >
> > I want to scrape some web pages and I am wondering if anyone here uses
> > J to do so? Any tips? Otherwise I'll have to resort to python.
> >
> > Thanks for any suggestions,
> > Ryan
> > ----------------------------------------------------------------------
> > For information about J forums see http://www.jsoftware.com/forums.htm
> >
> ----------------------------------------------------------------------
> For information about J forums see http://www.jsoftware.com/forums.htm
>
--
Devon McCormick, CFA
----------------------------------------------------------------------
For information about J forums see http://www.jsoftware.com/forums.htm
