You should start with the extension points that Nutch offers. These are
very similar to OSGI and Eclipse plug-ins.

Once you understand this, but  start writing your parse. Test and
implement.


Hope this helps.

Best regards,


Evert Wagenaar.

http://www.ejwagenaar.com/

On Mon, 12 Feb 2018 at 08:56 Yossi Tamari <yossi.tam...@pipl.com> wrote:

> The plug-in.includes order does not matter.
> To define the order of HtmlParseFilters, use the property
> htmlparsefilter.order.
> To define the order of Parsers, use the file conf/parse-plugins.xml. Note
> that once a single Parser returns a result, the following parsers will not
> be run.
>
> > -----Original Message-----
> > From: David Ferrero [mailto:david.ferr...@zion.com]
> > Sent: 12 February 2018 06:23
> > To: dev@nutch.apache.org
> > Subject: Re: Custom Parser / Indexer Starting points
> >
> > Thank you for all the tips. I think I need to understand better the
> pipeline of
> > parsers and if/how their plug-in.includes order  matters.
> >
> > > On Feb 11, 2018, at 1:18 AM, Yossi Tamari <yossi.tam...@pipl.com>
> wrote:
> > >
> > > Hi David,
> > >
> > > The interfaces related to extending Nutch parser/indexer are actually
> > > very simple. However, finding up-to-date documented samples is not.
> > > Luckily, Nutch comes with plenty built-in, so my suggestion would be
> > > to pick one, and dive into its implementation. Then just copy its
> > > folder and use it as a skeleton, replacing the specific logic (and
> plugin
> > metadata).
> > >
> > > The first question you need to ask yourself is if you really want to
> > > write a Parser/Indexer or just a HtmlParseFilter/IndexingFilter. I
> > > suspect that the default behaviour of the Nutch Parser and Indexer is
> > > useful for you, and you just want to add more functionality (that is
> > > what Any23 is doing). You can chain Filters, so your code could also
> > > leverage the Any23 logic, for example.
> > >
> > > The documentation starting point is the Wiki
> > > (https://wiki.apache.org/nutch/). For your specific question, this is
> > > the most relevant page: https://wiki.apache.org/nutch/AboutPlugins.
> > >
> > > One (old) example of writing a custom parser can be found here:
> > > http://www.treselle.com/blog/apache-nutch-with-custom-parser/. I
> > > suggest you Google for more information as needed, but always keep in
> > > mind that things may have changed over time.
> > >
> > > I think the best approach for domain-specific parsers is to have a
> > > custom parser that maps from the URL to the specific code. This can be
> > > just one big if/else, or a Map of domain->code (possibly using
> > > functional programming), or you can even have this map configurable in
> some
> > file.
> > >
> > > Once you have more specific questions/problems, I suggest you email
> > > u...@nutch.apache.org. dev@nutch.apache.org is intended for discussing
> > > code contributions to Nutch, as far as I understand, and I think less
> > > people see your messages here. (Also, more people will benefit from
> > > your questions
> > > there...)
> > >
> > > In summary, from my experience, writing any one of these plugins is
> > > really easy (discounting your own complex logic, of course), just
> > > implementing one or a few methods, changing some plugin XML file, and
> > > adding your extension to the global build (Ant) files. But to really
> > > understand how the passed data looks, and what you can do with it,
> > > debugging (in local mode) is the ultimate tool, and in the end is much
> > > more time-efficient than looking for information on the web. This is
> > > partly because a lot of the data is passed in Map-like form, so even
> > > the JavaDoc doesn't really tell you what will be there (it depends on
> > > what plugins you have configured, and how you configured those
> plugins...).
> > >
> > >    Yossi.
> > >
> > >
> > >> -----Original Message-----
> > >> From: David Ferrero [mailto:david.ferr...@zion.com]
> > >> Sent: 11 February 2018 04:00
> > >> To: dev@nutch.apache.org
> > >> Subject: Custom Parser / Indexer Starting points
> > >>
> > >> &tldr; If I wanted to learn about the nutch pipeline at a high level,
> > >> then
> > > write a
> > >> custom parser / indexer of my own where would a starting point be?
> > >>
> > >> I have used the latest 1.x Nutch to crawl a few specific websites and
> > >> been disappointed with the results, even after experimenting with new
> > >> html- microdata capabilities with updates to Any23 project
> > >> incorporated by
> > > Nutch, I
> > >> am still not (yet) excited. Bottom line is website data is not well
> > > structured and
> > >> not super friendly to algorithmic consumption (but you already knew
> that).
> > > To
> > >> that end, I am interested to developer custom parsers per internet
> > >> domain
> > > in an
> > >> effort to capture specific domain data. It currently looks like the
> > > plugin.includes
> > >> does not allow a per domain-based approach for parser / indexer. I
> > >> wonder
> > > if
> > >> someone could guide me toward a high level view of the Nutch data
> > > pipeline,
> > >> then guide me towards where to get started for creating custom
> > >> parsers
> > > that
> > >> might support a per-domain approach?
> > >>
> > >> Thanks,
> > >> David
> > >
>
> --
Sent from Gmail IPad

Reply via email to