Andrzej Bialecki wrote:
Dennis Kubes wrote:
I completely agree with this. I am interested in devoting as much
time as possible to seeing the success of Nutch, Hadoop, and Lucene.
As our business grows I would also be willing to devote developers
full time to work on Nutch, Hadoop, and Lucene.
I think that at least one company needs to come out and have a
production search engine that is competition, however small, to the
googles and yahoos of the world, built on Nutch and Hadoop. I thought
that was the original goal of Nutch. I know there are some out there
right now like Mozdex, but I mean a true billion page system. I think
the .8 codebase, and yes improvements could be made, is capable of
supporting such a system. I think then you will see many more
developers become interested in the project. If you build it they
will come.
Sure, I'd love to point people to such a system. But did you do a
calculation how much money in the initial investment, and then ongoing
costs, is needed to maintain such an index? It cannot happen just
because of someone's goodwill, there must be a sound business idea
behind it, and a team of dedicated people to make it happen and
persevere - not just to demonstrate how good Nutch is, but to keep up
for the sake of their own business.
I completely agree. We have been working on this business for almost a
year. We received significant seed capital to build the alpha version
of the search, which is complete, and are in the process of securing
first round private equity funding to scale to 100M pages this year and
up to 1B pages in year 2.
Yes the initial investment for hardware, data center costs, marketing
costs, and most importantly development staff for say a 1 billion page
index capable of supporting 100 queries per second constant is around 5M
and as it grows into the 10-20 billion range costs can grow as high as 100M.
I think what many people don't understand is that search is as much a
hardware (electricity, bandwidth) issue as it is a software issue. I
know that we couldn't have developed the systems we have without Nutch,
Hadoop, and Lucene and that I personally and we as a company are
completely committed to their development.
I will say that it is difficult for people to understand how to get
more involved. I have been working with Nutch and Hadoop for almost a
year now on a daily basis and only now am I understanding how to
contribute through jira, etc. There needs to be more guidance in
helping developers contribute. For example if you want to develop a
new piece of function they do x, y, and z. Here is how to patch your
system. If you want to develop a patch then here are the steps. I
have programmed in Java for many years but haven't worked on many open
source projects before. The process of how they work isn't explicit
and it needs to be.
Hmm. I might not be objective here anymore. There is however some
documentation already on the Wiki, which explains how to contribute - if
you feel it's inadequate please use your hard-earned experience to
improve it.
I am in the middle of writing a new wiki page for contributing that will
go into much more detail about the process.
We worked up many patches for issues we came up against in the .8 and
.4 codebases but they were never contributed because, as stupid as it
might sound, we really don't know how to give it back. The best thing
I thought I could do was to help answer questions on the list. Again
just need a little guidance.
Are you willing to spend the time and do the required refactoring?
Anyone else?
Yes, I am and I currently have 2 other developers that can help.
Sounds great. We could start by creating a new page on Wiki, which would
collect our vision for Nutch - as I mentioned to Stefan, I think we
should take a step back, and think about the strategy for the next 1-2
years of Nutch development, and what is the target audience.
I am all for this, just understand this is a new process for me so will
need some guidance.
Sure if we start a 2.x branch and if I'm not developing for the trash
or "jira nirvana", I can imaging to contribute. I
Just a quick comment: "jira nirvana" (which I believe refers to patches
sitting idle in Jira for a long time) is not caused by ill will or
disrespect for contributors, but foremost by limited human resources. If
we want to maintain a certain level of quality, these patches cannot be
applied blindly, but need to be reviewed, analyzed, applied, tested, and
committed. That's an awful lot of work for 2-3 people, who also have
other things to do ...
It is very less attractive to developers spending weeks to find a bug
like the regular expression one. Than such a bug sits there for month
in the jira being rejected. Sure if nobody of the contributors run
nutch with a 500 mio url
It's not being rejected - see the comments on that issue, there is an
overall agreement that it's ok; it simply hasn't been applied yet. See
above for the why.
I'm slowly coming to a point where I should be able to fix it - but
let's not throw out the baby with the water ...
Wow, I hold my finger crossed!
There is a great book on this. It is 0691122024. Andrzej send me
your address and I will buy and ship you a copy if you don't have it.
Too late :) I found it two weeks ago, and it's already on its merry way
- but thanks for the offer.
We would also be willing to help develop this functionality further.
I started working on a testbed as a part of another commercial project,
it's likely that I could get a release from the customer to contribute
this code to the project. A testbed is a prerequisite for any serious
work on ranking and web graph.
(It's quite unfortunate that the best-of-breed open source framework for
working with web graphs is licensed under LGPL ...)
I can definitely see a desire to re-write but I think even if you
re-write you are still going to have the same problem. Search is hard
and without guidance we can't get enough developers to understand what
they need to know to help.
Indeed. People often don't appreciate how much heuristics and trials,
beyond pure academic-level IR, is needed to come up with a system that
gives a decent quality of results, and is manageable. Nutch may not be
perfect, but there's a lot of this specific knowledge already
accumulated here.
Absolutely and it is not knowledge that is easily found elsewhere.
At this time I don't think it is a design problem I think it is a
people problem. I will be more than willing to head up training,
documenting, and helping developers get up to speed. I just need
direction in this area myself.
I believe that at this point it's crucial to keep the project
well-focused (at the moment I think the main focus is on larger
installations, and not the small ones), and also to make Nutch
attractive to developers as a reusable "search engine" component.
I think there are two areas. One is to keep the focus as you stated
above. The other is to provide a path to get more people involved. If
no one objects I will continue working on such a path.
Let's continue the discussion. I'll create the page on Wiki, please feel
free to add your thoughts.
Will do.
