Re: Have anybody thought of replacing CrawlDb with any kind of Rational DB?

2007-04-17 Thread wangxu
Andrzej Bialecki wrote: Howie Wang wrote: I definitely don't expect people to write it just because it happens to be useful to me :-) Call me crazy, but I'm thinking of implementing this when I get some free time (whenever that will be). It seems that I would just need to implement

Re: Have anybody thought of replacing CrawlDb with any kind of Rational DB?

2007-04-13 Thread Andrzej Bialecki
Howie Wang wrote: Sorry about the previous crappily formatted message. In brief, my point wasthat relational DB might perform better for small niche users, and plusyou get the flexibility of SQL. No more writing custom code to tweak webdb.Howie Generally speaking, I agree that it would be a

RE: Have anybody thought of replacing CrawlDb with any kind of Rational DB?

2007-04-13 Thread Howie Wang
I definitely don't expect people to write it just because it happens to be useful to me :-) Call me crazy, but I'm thinking of implementing this when I get some free time (whenever that will be). It seems that I would just need to implement IWebDBWriter and IWebDBReader, and then add a

Re: Have anybody thought of replacing CrawlDb with any kind of Rational DB?

2007-04-13 Thread Arun Kaundal
Actually nutch people are kind of autocrate., don't expect more from them They do what they have decided I am waiting really stable product with incremental indexing, which detect and add/remove pages as soon as they added/removed. But they don't want to this , i don't know why ? what is

Re: Have anybody thought of replacing CrawlDb with any kind of Rational DB?

2007-04-13 Thread Doug Cutting
Arun Kaundal wrote: Actually nutch people are kind of autocrate., don't expect more from them They do what they have decided Have you submitted patches that have been ignored or rejected? Each Nutch contributor indeed does what he or she decides. Nutch is not a service organization that

Re: Have anybody thought of replacing CrawlDb with any kind of Rational DB?

2007-04-13 Thread Andrzej Bialecki
Howie Wang wrote: I definitely don't expect people to write it just because it happens to be useful to me :-) Call me crazy, but I'm thinking of implementing this when I get some free time (whenever that will be). It seems that I would just need to implement IWebDBWriter and IWebDBReader, and

RE: Have anybody thought of replacing CrawlDb with any kind of Rational DB?

2007-04-13 Thread Howie Wang
Thanks for the input, Andrzej. Yes, I'm still working off of 0.7. I might still try it since I'm not planning on upgrading for a while, but it sounds like it's not going to port to the current versions. Howie _ Your friends are

Have anybody thought of replacing CrawlDb with any kind of Rational DB?

2007-04-12 Thread wangxu
Have anybody thought of replacing CrawlDb with any kind of Rational DB,mysql,for example? Crawldb is so difficult to manipulate. I often have the requirements to edit several entries in crawdb; But that would cost too much waiting for the mapReduce.

Re: Have anybody thought of replacing CrawlDb with any kind of Rational DB?

2007-04-12 Thread Nuther
Hi, wangxu. You wrote 13 апреля 2007 г., 1:03:31: Have anybody thought of replacing CrawlDb with any kind of Rational DB,mysql,for example? Crawldb is so difficult to manipulate. I often have the requirements to edit several entries in crawdb; But that would cost too much waiting for the

Re: Have anybody thought of replacing CrawlDb with any kind of Rational DB?

2007-04-12 Thread Andrzej Bialecki
wangxu wrote: Have anybody thought of replacing CrawlDb with any kind of Rational DB,mysql,for example? Crawldb is so difficult to manipulate. I often have the requirements to edit several entries in crawdb; But that would cost too much waiting for the mapReduce. Please make the following

Re: Have anybody thought of replacing CrawlDb with any kind of Rational DB?

2007-04-12 Thread Sami Siren
wangxu wrote: Have anybody thought of replacing CrawlDb with any kind of Rational DB,mysql,for example? Crawldb is so difficult to manipulate. I often have the requirements to edit several entries in crawdb; But that would cost too much waiting for the mapReduce. Once when I was young

Re: Have anybody thought of replacing CrawlDb with any kind of Rational DB?

2007-04-12 Thread Dennis Kubes
Andrzej Bialecki wrote: wangxu wrote: Have anybody thought of replacing CrawlDb with any kind of Rational DB,mysql,for example? Crawldb is so difficult to manipulate. I often have the requirements to edit several entries in crawdb; But that would cost too much waiting for the mapReduce.

RE: Have anybody thought of replacing CrawlDb with any kind of Rational DB?

2007-04-12 Thread Howie Wang
Please make the following test using your favorite relational DB:* create a table with 300 mln rows and 10 columns of mixed type* select 1 mln rows, sorted by some value* update 1 mln rows to different valuesIf you find that these operations take less time than with

RE: Have anybody thought of replacing CrawlDb with any kind of Rational DB?

2007-04-12 Thread Howie Wang
Sorry about the previous crappily formatted message. In brief, my point wasthat relational DB might perform better for small niche users, and plusyou get the flexibility of SQL. No more writing custom code to tweak webdb.Howie _