Re: NUTCH-1129, Any23, microdata parsing, indexing, and extraction?

[David Ferrero]

Just to be clear I'm using any23 in the plugin.includes, I am getting 
Any23-Triples metadata. However I am hoping to see more Any23-Triples when I 
added json+ld extractors to any23.extractors... 

./bin/nutch parsechecker 
https://job-openings.monster.com/Senior-Software-Engineer-iOS-Boulder-CO-US-Housecanary/31/4edf245d-e563-43c2-9eca-e6c7c891d17a

fetching: 
https://job-openings.monster.com/Senior-Software-Engineer-iOS-Boulder-CO-US-Housecanary/31/4edf245d-e563-43c2-9eca-e6c7c891d17a
robots.txt whitelist not configured.
parsing: 
https://job-openings.monster.com/Senior-Software-Engineer-iOS-Boulder-CO-US-Housecanary/31/4edf245d-e563-43c2-9eca-e6c7c891d17a
contentType: text/html
signature: 84dcab335b0ae039c612dc8945bb9853
-
Url
---

https://job-openings.monster.com/Senior-Software-Engineer-iOS-Boulder-CO-US-Housecanary/31/4edf245d-e563-43c2-9eca-e6c7c891d17a
-
ParseData
-

Version: 5
Status: success(1,0)
Title: Senior Software Engineer - iOS job at Housecanary | Monster.com
Outlinks: 1
  outlink: toUrl: 
https://job-openings.monster.com/Senior-Software-Engineer-iOS-Boulder-CO-US-Housecanary/31/4edf245d-e563-43c2-9eca-e6c7c891d17a
 anchor: 
Content Metadata: Server=Microsoft-IIS/8.5 Access-Control-Allow-Origin=* 
Access-Control-Allow-Methods=PUT, GET, POST Connection=Close Pragma=no-cache 
Access-Control-Allow-Headers=Origin, X-Requested-With, Content-Type, Accept 
Date=Fri, 09 Feb 2018 07:42:02 GMT nutch.crawl.score=0.0 
X-AspNetMvc-Version=5.2 nutch.fetch.time=1518162123187 
X-Frame-Options=SAMEORIGIN X-UA-Compatible=IE=edge Cache-Control=no-cache, 
no-store Content-Encoding=gzip X-AspNet-Version=4.0.30319 
Set-Cookie=atmResolver=|Seeker|jobviewcloud|58|164|; path=/; HttpOnly 
Expires=-1 Content-Length=37894 Content-Type=text/html; charset=utf-8 
X-Powered-By=ASP.NET 
Parse Metadata: CharEncodingForConversion=utf-8 OriginalCharEncoding=utf-8 
Any23-Triples= viewport=user-scalable=no, width=device-width, initial-scale=1, 
minimum-scale=1, maximum-scale=1 description==Complete the job application for 
Senior Software Engineer - iOS in Boulder, CO online today or find more job 
listings available at Housecanary at Monster. robots=all 
format-detection=telephone=no 

Here is the application/ld+json in that job page that I was hoping would get 
extracted with Any23...


{"title":"Senior Software Engineer - 
iOS","datePosted":"2017-07-27","description":"
At HouseCanary, we’re 
using data and analytics to predict the future of US residential real estate. 
Our goal is to help people make better decisions by offering innovative and 
unparalleled insights. HouseCanary’s platform accurately forecasts values 36 
months into the future for three million residential blocks and more than 100 
million properties.
We're seeking a passionate Senior iOS Engineer to 
help build our disruptive mobile products.
What you'll 
do:
Integrate maps and location tracking (driving 
directions, optimized routes, etc.)
Refine our Sketch feature used to design 
floor plans
Work with heavy core data
Camera integration
And 
more!
What you have:
Expertise in Swift and 
Objective-C
Experience with Cocoa Touch, UIKit, Core Animation, Core 
Graphics, Core Location, MapKit, etc.
Autolayout and Storyboards 
exposure
Core Data knowledge
Familiarity with multi-threading and 
GCD
Experience interfacing with RESTful JSON APIs
Bonus 
points for knowledge of:
Real estate markets
Predictive 
modeling
HouseCanary is the authoritative source for accurate, uniform 
information, analyzed and visualized real-time to make better, faster 
decisions.
HouseCanary - see into the future of real 
estate.
","educationRequirements":"Not 
specified","jobLocation":{"address":{"addressLocality":"Boulder","addressRegion":"CO","addressCountry":"US","@type":"PostalAddress"},"@type":"Place"},"hiringOrganization":{"name":"Housecanary","@type":"Organization"},"experienceRequirements":"Not
 
specified","identifier":{"name":"Housecanary","value":"202874","@type":"PropertyValue"},"@context":"http://schema.org","@type":"JobPosting"}



> On Feb 9, 2018, at 12:31 AM, David Ferrero  wrote:
> 
> Thank you for this information. Since this is very much related to Any23 and 
> microdata parsing, I’m going to ask what I believe is a related question but 
> keep this same thread so it will be organized in one place:
> 
> I noticed a lot of job boards such as dice.com, monster.com, etc use 
> http://schema.org/JobPosting information, however many seem to use  type="application/ld+json”>… rather than RDF.
> Summer 2017, Google announced structured data guidance for Jobs:
> https://developers.google.com/search/docs/data-types/job-posting
> and a testing tool to validate your 

Re: NUTCH-1129, Any23, microdata parsing, indexing, and extraction?

[David Ferrero]

Thank you for this information. Since this is very much related to Any23 and 
microdata parsing, I’m going to ask what I believe is a related question but 
keep this same thread so it will be organized in one place:

I noticed a lot of job boards such as dice.com , monster.com 
, etc use http://schema.org/JobPosting 
 information, however many seem to use 

Re: NUTCH-1129, Any23, microdata parsing, indexing, and extraction?

[lewis john mcgibbney]

Hi David,
Answers inline

On Thu, Feb 8, 2018 at 9:19 AM,  wrote:

>
> From: David Ferrero 
> To: user@nutch.apache.org
> Cc:
> Bcc:
> Date: Thu, 8 Feb 2018 10:19:52 -0700
> Subject: NUTCH-1129, Any23, microdata parsing, indexing, and extraction?
> Pull request #205 was recently merged into master branch for Nutch 1.x in
> fulfillment of NUTCH-1129 "microdata for Nutch 1.x"
>
> I am new to nutch and solr and have just started crawling and indexing a
> few select websites. Using the built in html parsing/indexing, I am getting
> searchable fields like url, content, host, sometimes a title, and a few
> other indexing related fields like digest, boost, segment, and tstamp. That
> said, I realized very quickly that I need better results. While exploring
> the source of the website, I noticed references to schema.org and get
> excited by what I see. That’s how I stumbled upon NUTCH-1129.
>
> I’ve built apache-nutch-1.15-SNAPSHOT which includes Any23 parser/indexer.
>

Excellent.


>
> Q: Now what?  How do I gain Any23 microdata parsing / indexing
> capabilities introduced by NUTCH-1129?
> Q: Do I replace parse-(html | tika)|index-(basic | anchor) in
> plugin.includes with something like parse-(html | tika |
> any23)|index-(basic | anchor | any23)
>

No, you just add 'any23' to the list of plugins within the plugin.includes
property of nutch-site.xml


> Q: How do I expose the discovered microdata structure / items to end-user
> such as Solr? For example, what are the microdata items and do I need to
> map them to Solr in solrindex-mapping.xml?
>

OK, so current configuration for the Any23 plugin, is to store extracted
structured data markup in the Nutch Metadata object with a key "
Any23-Triples". You can locate it using something like the ParserChekcer
tool provided via the 'nutch' script. Liekwise you can also locate it, as a
representation of what would be indexed, by using the IndexerChecker
tooling also provided within the 'nutch' script.

An example would be as follows, data is now indexed as follows (example
after crawling https://smartive.ch/jobs):


  "structured_data": [
{
  "node": "",
  "value": "\"IE-edge,chrome=1\"@de",
  "key": "",
  "short_key": "X-UA-Compatible"
},
{
  "node": "",
  "value": "\"Wir sind smartive \\u2014 eine dynamische,
innovative Schweizer Webentwicklungsagentur. Die Realisierung
zeitgem\\u00E4sser Webl\\u00F6sungen geh\\u00F6rt genauso zu unserer
Passion, wie die konstruktive Zusammenarbeit mit unseren Kundinnen und
Kunden.\"@de",
  "key": "",
  "short_key": "description"
},
{
  "node": "",
  "value": "\"width=device-width, initial-scale=1,
shrink-to-fit=no\"@de",
  "key": "",
  "short_key": "viewport"
},
{
  "node": "",
  "value": "\"width=device-width,initial-scale=1\"@de",
  "key": "",
  "short_key": "viewport"
},
{
  "node": "",
  "value": "\"ie=edge\"@de",
  "key": "",
  "short_key": "x-ua-compatible"
}
  ],


Note from above, that the 'predicate' key field is very useful for quickly
filtering through, for example, Hotel Ratings, or something similar.


>
> I’d also be interested to learn how to point at a specific URL and see how
> nutch sees the microdata (best case), then learn how to leverage this into
> nutch and finally into solr.
>
>
See the tooling for ParserChecker and IndexerChecker as explained above.
Any further question, please let me know.
Lewis

NUTCH-1129, Any23, microdata parsing, indexing, and extraction?

[David Ferrero]

Pull request #205 was recently merged into master branch for Nutch 1.x in 
fulfillment of NUTCH-1129 "microdata for Nutch 1.x"

I am new to nutch and solr and have just started crawling and indexing a few 
select websites. Using the built in html parsing/indexing, I am getting 
searchable fields like url, content, host, sometimes a title, and a few other 
indexing related fields like digest, boost, segment, and tstamp. That said, I 
realized very quickly that I need better results. While exploring the source of 
the website, I noticed references to schema.org and get excited by what I see. 
That’s how I stumbled upon NUTCH-1129.

I’ve built apache-nutch-1.15-SNAPSHOT which includes Any23 parser/indexer. 

Q: Now what?  How do I gain Any23 microdata parsing / indexing capabilities 
introduced by NUTCH-1129? 
Q: Do I replace parse-(html | tika)|index-(basic | anchor) in plugin.includes 
with something like parse-(html | tika | any23)|index-(basic | anchor | any23)
Q: How do I expose the discovered microdata structure / items to end-user such 
as Solr? For example, what are the microdata items and do I need to map them to 
Solr in solrindex-mapping.xml?

I’d also be interested to learn how to point at a specific URL and see how 
nutch sees the microdata (best case), then learn how to leverage this into 
nutch and finally into solr. 

Thanks for any guidance.
-David