Hi,
Well I still get a very slow mergesegs:
[EMAIL PROTECTED] nutch]# tail -f nutch-mergesegs-kunzon.com.log
050919 171351 Processed 120000 records (1146.5918 rec/s)
050919 171408 Processed 140000 records (1158.2788 rec/s)
050919 171428 Processed 160000 records (1019.8358 rec/s)
050919 171451 Processed 180000 records (879.2368 rec/s)
050919 171510 Processed 200000 records (1054.9636 rec/s)
050919 171528 Processed 220000 records (1069.2328 rec/s)
050919 171547 Processed 240000 records (1099.868 rec/s)
050919 171832 - creating next subindex...
050919 174512 Processed 260000 records (11.328647 rec/s)
050919 200315 Processed 280000 records (2.4145627 rec/s)
It is falling to 2.4 res per second ...
Can somebody help me please. 400K records is only the beginning what
will happen when it is 4M?
Regards,
Gal
EM wrote:
'segslice' all partial segments into new ones prior merging.
Gal Nitzan wrote:
Hi Doug,
Thank you for the prompt reply.
Well things got much much faster (i guess about 40% faster), but it
seems that something got really corrupted. Everything gets stuck
after 40K records.
>050917 043332 - data in segment index/segments/20050916014401 is
corrupt, using only 128115 entries.
.
-------------------------------------------------------
SF.Net email is sponsored by:
Tame your development challenges with Apache's Geronimo App Server.
Download it for free - -and be entered to win a 42" plasma tv or your very
own Sony(tm)PSP. Click here to play: http://sourceforge.net/geronimo.php
_______________________________________________
Nutch-general mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/nutch-general
