I'm trying to reindex my segments on a new elasticsearch server, and I'm having
trouble. Sometimes, a segment will get indexed fine, but then on the next
segment it will fail. I'm not seeing anything in elasticsearch's logs that
would indicate a problem on that end (but I'm admittedly way out of area of
expertise in dealing with this stuff0.
Below is what I'm seeing in nutch's hadoop.log. This is a fresh log file (I
deleted the old one before running the bin/nutch index command). In this case
it made it part way through indexing the segment before failing (I was watching
the document count increase in marvel). Below that is the elasticsesarch log
for the same timeframe.
Any idea what I might be doing wrong or how I might go about diagnosing the
issue? Thanks,
Jeff Jackson
Hadoop.log:
2015-10-13 16:44:40,533 INFO indexer.IndexingJob - Indexer: starting at
2015-10-13 16:44:40
2015-10-13 16:44:40,645 INFO indexer.IndexingJob - Indexer: deleting gone
documents: false
2015-10-13 16:44:40,645 INFO indexer.IndexingJob - Indexer: URL filtering:
false
2015-10-13 16:44:40,645 INFO indexer.IndexingJob - Indexer: URL normalizing:
false
2015-10-13 16:44:40,919 INFO indexer.IndexWriters - Adding
org.apache.nutch.indexwriter.elastic.ElasticIndexWriter
2015-10-13 16:44:40,920 INFO indexer.IndexingJob - Active IndexWriters :
ElasticIndexWriter
elastic.cluster : elastic prefix cluster
elastic.host : hostname
elastic.port : port
elastic.index : elastic index command
elastic.max.bulk.docs : elastic bulk index doc counts. (default 250)
elastic.max.bulk.size : elastic bulk index length. (default 2500500
~2.5MB)
2015-10-13 16:44:40,922 INFO indexer.IndexerMapReduce - IndexerMapReduce:
crawldb: /root/apache-nutch-1.10/crawl/crawldb
2015-10-13 16:44:40,922 INFO indexer.IndexerMapReduce - IndexerMapReduce:
linkdb: /root/apache-nutch-1.10/crawl/linkdb
2015-10-13 16:44:40,923 INFO indexer.IndexerMapReduce - IndexerMapReduces:
adding segment: /root/apache-nutch-1.10/crawl/segments/20150526191748
2015-10-13 16:44:41,032 WARN util.NativeCodeLoader - Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
2015-10-13 16:44:41,695 INFO anchor.AnchorIndexingFilter - Anchor
deduplication is: off
2015-10-13 16:46:59,229 INFO indexer.IndexWriters - Adding
org.apache.nutch.indexwriter.elastic.ElasticIndexWriter
2015-10-13 16:46:59,339 INFO elasticsearch.plugins - [Grandmaster] loaded [],
sites []
2015-10-13 16:47:01,579 INFO elastic.ElasticIndexWriter - Processing bulk
request [docs = 205, length = 2519186, total docs = 205, last doc in bulk =
'http://3forjc.blogspot.com/2010_11_01_archive.html']
2015-10-13 16:47:01,998 INFO elastic.ElasticIndexWriter - Processing bulk
request [docs = 63, length = 2594569, total docs = 268, last doc in bulk =
'http://4womaninthewilderness.blogspot.com/2013_02_01_archive.html']
2015-10-13 16:47:02,170 INFO elastic.ElasticIndexWriter - Previous took in ms
384, including wait 171
2015-10-13 16:47:02,381 INFO elastic.ElasticIndexWriter - Processing bulk
request [docs = 106, length = 2506430, total docs = 374, last doc in bulk =
'http://5621582817745579273_47da371f54bd3f164898f6392f5bdadc3d86df5e.blogspot.com/2015/05/vatican-officially-recognizes-state-of.html']
2015-10-13 16:47:02,824 INFO elastic.ElasticIndexWriter - Previous took in ms
541, including wait 443
2015-10-13 16:47:03,109 INFO elastic.ElasticIndexWriter - Processing bulk
request [docs = 230, length = 2510289, total docs = 604, last doc in bulk =
'http://abc3miscellany.blogspot.com/2015_02_01_archive.html']
2015-10-13 16:47:03,622 INFO elastic.ElasticIndexWriter - Previous took in ms
604, including wait 513
2015-10-13 16:47:03,884 INFO elastic.ElasticIndexWriter - Processing bulk
request [docs = 250, length = 1938835, total docs = 854, last doc in bulk =
'http://activemindbodyandsoul.org/category/daily-climb/']
2015-10-13 16:47:04,287 INFO elastic.ElasticIndexWriter - Previous took in ms
610, including wait 403
2015-10-13 16:47:04,485 INFO elastic.ElasticIndexWriter - Processing bulk
request [docs = 174, length = 2502740, total docs = 1028, last doc in bulk =
'http://aglow.com/resources/leader-development/prophetic-messages']
2015-10-13 16:47:05,089 INFO elastic.ElasticIndexWriter - Previous took in ms
713, including wait 604
2015-10-13 16:47:05,215 INFO elastic.ElasticIndexWriter - Processing bulk
request [docs = 250, length = 1347205, total docs = 1278, last doc in bulk =
'http://aglowinternational.org/give/a-company']
2015-10-13 16:47:05,867 INFO elastic.ElasticIndexWriter - Previous took in ms
718, including wait 652
2015-10-13 16:47:06,126 INFO elastic.ElasticIndexWriter - Processing bulk
request [docs = 250, length = 2298233, total docs = 1528, last doc in bulk =
'http://allsoulschristianchurch.com/mediaPlayer/']
2015-10-13 16:47:06,126 INFO elastic.ElasticIndexWriter - Previous took in ms
198, including wait 0
2015-10-13 16:47:06,270 INFO elastic.ElasticIndexWriter - Processing bulk
request [docs = 195, length = 2509543, total docs = 1723, last doc in bulk =
'http://amazingfactsministries.com/index.php/publications/online-library/life-in-the-spirit']
2015-10-13 16:47:06,471 INFO elastic.ElasticIndexWriter - Previous took in ms
296, including wait 201
2015-10-13 16:47:06,654 INFO elastic.ElasticIndexWriter - Processing bulk
request [docs = 92, length = 2563300, total docs = 1815, last doc in bulk =
'http://ancientchristiandefender.blogspot.com/2008_06_01_archive.html']
2015-10-13 16:47:07,069 INFO elastic.ElasticIndexWriter - Previous took in ms
544, including wait 414
2015-10-13 16:47:07,186 INFO elastic.ElasticIndexWriter - Processing bulk
request [docs = 94, length = 2502386, total docs = 1909, last doc in bulk =
'http://andreayorkmuse.blogspot.com/2013_11_01_archive.html']
2015-10-13 16:47:07,650 INFO elastic.ElasticIndexWriter - Previous took in ms
461, including wait 463
2015-10-13 16:47:07,873 INFO elastic.ElasticIndexWriter - Processing bulk
request [docs = 250, length = 1096352, total docs = 2159, last doc in bulk =
'http://anunworthyservant.com/tag/churchianity/']
2015-10-13 16:47:08,026 INFO elastic.ElasticIndexWriter - Previous took in ms
320, including wait 153
2015-10-13 16:47:08,131 INFO elastic.ElasticIndexWriter - Processing bulk
request [docs = 98, length = 2551881, total docs = 2257, last doc in bulk =
'http://apocalypse2010.blogspot.com/2012_12_01_archive.html']
2015-10-13 16:47:08,228 INFO elastic.ElasticIndexWriter - Previous took in ms
178, including wait 97
2015-10-13 16:47:08,424 INFO elastic.ElasticIndexWriter - Processing bulk
request [docs = 199, length = 2513815, total docs = 2456, last doc in bulk =
'http://apostolicendtimescenario.blogspot.com/2009/08/is-third-temple-legitimate.html']
2015-10-13 16:47:13,989 INFO elastic.ElasticIndexWriter - Previous took in ms
5687, including wait 5565
2015-10-13 16:47:14,113 INFO elastic.ElasticIndexWriter - Processing bulk
request [docs = 138, length = 2505616, total docs = 2594, last doc in bulk =
'http://apostolicvision.blogspot.com/2010/02/toxicology-of-complaining.html']
2015-10-13 16:47:14,339 INFO elastic.ElasticIndexWriter - Previous took in ms
284, including wait 226
2015-10-13 16:47:14,689 INFO elastic.ElasticIndexWriter - Processing bulk
request [docs = 241, length = 2524444, total docs = 2835, last doc in bulk =
'http://armstrongismlibrary.blogspot.ca/2013_10_06_archive.html']
2015-10-13 16:47:14,811 INFO elastic.ElasticIndexWriter - Previous took in ms
416, including wait 121
2015-10-13 16:47:14,898 INFO elastic.ElasticIndexWriter - Processing bulk
request [docs = 37, length = 2504364, total docs = 2872, last doc in bulk =
'http://armstrongismlibrary.blogspot.ca/2014_06_22_archive.html']
2015-10-13 16:47:15,411 INFO elastic.ElasticIndexWriter - Previous took in ms
544, including wait 513
2015-10-13 16:47:15,506 INFO elastic.ElasticIndexWriter - Processing bulk
request [docs = 43, length = 2535017, total docs = 2915, last doc in bulk =
'http://armstrongismlibrary.blogspot.ca/2015_04_19_archive.html']
2015-10-13 16:47:15,869 INFO elastic.ElasticIndexWriter - Previous took in ms
402, including wait 363
2015-10-13 16:47:15,964 INFO elastic.ElasticIndexWriter - Processing bulk
request [docs = 36, length = 2527853, total docs = 2951, last doc in bulk =
'http://armstrongismlibrary.blogspot.co.nz/2014_04_20_archive.html']
2015-10-13 16:47:16,302 INFO elastic.ElasticIndexWriter - Previous took in ms
349, including wait 338
2015-10-13 16:47:16,393 INFO elastic.ElasticIndexWriter - Processing bulk
request [docs = 44, length = 2549719, total docs = 2995, last doc in bulk =
'http://armstrongismlibrary.blogspot.co.nz/2015_02_22_archive.html']
2015-10-13 16:47:23,374 INFO elastic.ElasticIndexWriter - Previous took in ms
7002, including wait 6981
2015-10-13 16:47:23,475 INFO elastic.ElasticIndexWriter - Processing bulk
request [docs = 36, length = 2567438, total docs = 3031, last doc in bulk =
'http://armstrongismlibrary.blogspot.co.uk/2014_02_23_archive.html']
2015-10-13 16:47:23,994 INFO elastic.ElasticIndexWriter - Previous took in ms
493, including wait 519
2015-10-13 16:47:24,083 INFO elastic.ElasticIndexWriter - Processing bulk
request [docs = 43, length = 2512172, total docs = 3074, last doc in bulk =
'http://armstrongismlibrary.blogspot.co.uk/2014_12_21_archive.html']
2015-10-13 16:47:25,074 INFO elastic.ElasticIndexWriter - Previous took in ms
948, including wait 991
2015-10-13 16:47:25,171 INFO elastic.ElasticIndexWriter - Processing bulk
request [docs = 37, length = 2511370, total docs = 3111, last doc in bulk =
'http://armstrongismlibrary.blogspot.com.au/2013_12_29_archive.html']
2015-10-13 16:47:25,767 INFO elastic.ElasticIndexWriter - Previous took in ms
597, including wait 596
2015-10-13 16:47:25,861 INFO elastic.ElasticIndexWriter - Processing bulk
request [docs = 41, length = 2535222, total docs = 3152, last doc in bulk =
'http://armstrongismlibrary.blogspot.com.au/2014_10_12_archive.html']
2015-10-13 16:47:26,902 INFO elastic.ElasticIndexWriter - Previous took in ms
1070, including wait 1041
2015-10-13 16:47:26,994 INFO elastic.ElasticIndexWriter - Processing bulk
request [docs = 41, length = 2524383, total docs = 3193, last doc in bulk =
'http://armstrongismlibrary.blogspot.com/2013_12_01_archive.html']
2015-10-13 16:47:28,314 INFO elastic.ElasticIndexWriter - Previous took in ms
1185, including wait 1319
2015-10-13 16:47:28,405 INFO elastic.ElasticIndexWriter - Processing bulk
request [docs = 41, length = 2532087, total docs = 3234, last doc in bulk =
'http://armstrongismlibrary.blogspot.com/2014_09_14_archive.html']
2015-10-13 16:47:29,500 INFO elastic.ElasticIndexWriter - Previous took in ms
1076, including wait 1095
2015-10-13 16:47:29,642 INFO elastic.ElasticIndexWriter - Processing bulk
request [docs = 70, length = 2517694, total docs = 3304, last doc in bulk =
'http://ask.yuriyandinna.com/category/relationships/finding-a-spouse/']
2015-10-13 16:47:30,128 INFO elastic.ElasticIndexWriter - Previous took in ms
513, including wait 486
2015-10-13 16:47:30,353 INFO elastic.ElasticIndexWriter - Processing bulk
request [docs = 250, length = 2232716, total docs = 3554, last doc in bulk =
'http://babyloniansquirrel.blogspot.com/2010_05_01_archive.html']
2015-10-13 16:47:31,097 INFO elastic.ElasticIndexWriter - Previous took in ms
895, including wait 743
2015-10-13 16:47:31,223 INFO elastic.ElasticIndexWriter - Processing bulk
request [docs = 118, length = 2518733, total docs = 3672, last doc in bulk =
'http://backtoluther.blogspot.com/2013_08_01_archive.html']
2015-10-13 16:47:31,771 INFO elastic.ElasticIndexWriter - Previous took in ms
573, including wait 548
2015-10-13 16:47:31,909 INFO elastic.ElasticIndexWriter - Processing bulk
request [docs = 74, length = 2515412, total docs = 3746, last doc in bulk =
'http://baptist-distinctives.blogspot.com/2009/02/verbal-and-plenary-inspiration-of-bible.html']
2015-10-13 16:47:32,864 INFO elastic.ElasticIndexWriter - Previous took in ms
1035, including wait 955
2015-10-13 16:47:32,962 INFO elastic.ElasticIndexWriter - Processing bulk
request [docs = 72, length = 2504157, total docs = 3818, last doc in bulk =
'http://baptist-rp.blogspot.com/2010/03/free-pdf-book-facebook-as-ministry-tool.html']
2015-10-13 16:47:33,588 INFO elastic.ElasticIndexWriter - Previous took in ms
540, including wait 626
2015-10-13 16:47:33,816 INFO elastic.ElasticIndexWriter - Processing bulk
request [docs = 250, length = 2417618, total docs = 4068, last doc in bulk =
'http://bearvalleychurch.org/slavic-gospel-the-mocks']
2015-10-13 16:47:34,653 INFO elastic.ElasticIndexWriter - Previous took in ms
881, including wait 836
2015-10-13 16:47:34,933 INFO elastic.ElasticIndexWriter - Processing bulk
request [docs = 245, length = 2595875, total docs = 4313, last doc in bulk =
'http://bethanylutheranworship.blogspot.com/2008_11_01_archive.html']
2015-10-13 16:47:35,136 INFO elastic.ElasticIndexWriter - Previous took in ms
431, including wait 203
2015-10-13 16:47:35,212 INFO elastic.ElasticIndexWriter - Processing bulk
request [docs = 17, length = 2515581, total docs = 4330, last doc in bulk =
'http://bethanylutheranworship.blogspot.com/2010_04_01_archive.html']
2015-10-13 16:47:35,940 INFO elastic.ElasticIndexWriter - Previous took in ms
746, including wait 728
2015-10-13 16:47:36,015 INFO elastic.ElasticIndexWriter - Processing bulk
request [docs = 15, length = 2521102, total docs = 4345, last doc in bulk =
'http://bethanylutheranworship.blogspot.com/2011_07_01_archive.html']
2015-10-13 16:47:36,428 INFO elastic.ElasticIndexWriter - Previous took in ms
433, including wait 412
2015-10-13 16:47:36,511 INFO elastic.ElasticIndexWriter - Processing bulk
request [docs = 19, length = 2548547, total docs = 4364, last doc in bulk =
'http://bethanylutheranworship.blogspot.com/2013_01_01_archive.html']
2015-10-13 16:47:37,171 INFO elastic.ElasticIndexWriter - Previous took in ms
687, including wait 660
2015-10-13 16:47:37,340 INFO elastic.ElasticIndexWriter - Processing bulk
request [docs = 40, length = 2502627, total docs = 4404, last doc in bulk =
'http://bible-truths-revealed.com/RevelationOutline.html']
2015-10-13 16:47:37,674 INFO elastic.ElasticIndexWriter - Previous took in ms
399, including wait 334
2015-10-13 16:47:39,044 INFO elastic.ElasticIndexWriter - Processing bulk
request [docs = 238, length = 2536423, total docs = 4642, last doc in bulk =
'http://biblenews1.com/grace/graced.htm']
2015-10-13 16:47:39,044 INFO elastic.ElasticIndexWriter - Previous took in ms
613, including wait 0
2015-10-13 16:47:39,317 INFO elastic.ElasticIndexWriter - Processing bulk
request [docs = 246, length = 2502515, total docs = 4888, last doc in bulk =
'http://biblicalcreationandevangelism.blogspot.com/2015_02_01_archive.html']
2015-10-13 16:47:39,851 INFO elastic.ElasticIndexWriter - Previous took in ms
751, including wait 533
2015-10-13 16:47:40,158 INFO elastic.ElasticIndexWriter - Processing bulk
request [docs = 163, length = 2583662, total docs = 5051, last doc in bulk =
'http://blog.chriskrycho.com/2010_10_01_archive.html']
2015-10-13 16:47:40,779 INFO elastic.ElasticIndexWriter - Previous took in ms
824, including wait 620
2015-10-13 16:47:41,056 INFO elastic.ElasticIndexWriter - Processing bulk
request [docs = 176, length = 2505011, total docs = 5227, last doc in bulk =
'http://blog.poweredby4.org/challenge/2012/01/']
2015-10-13 16:47:41,550 INFO elastic.ElasticIndexWriter - Previous took in ms
527, including wait 494
2015-10-13 16:47:41,797 INFO elastic.ElasticIndexWriter - Processing bulk
request [docs = 225, length = 2554774, total docs = 5452, last doc in bulk =
'http://bloggingscripturehisway.blogspot.com/2012_04_01_archive.html']
2015-10-13 16:47:42,308 INFO elastic.ElasticIndexWriter - Previous took in ms
702, including wait 510
2015-10-13 16:47:42,400 INFO elastic.ElasticIndexWriter - Processing bulk
request [docs = 24, length = 2632288, total docs = 5476, last doc in bulk =
'http://bloggingscripturehisway.blogspot.com/2014_02_01_archive.html']
2015-10-13 16:47:42,881 INFO elastic.ElasticIndexWriter - Previous took in ms
463, including wait 480
2015-10-13 16:47:43,012 INFO elastic.ElasticIndexWriter - Processing bulk
request [docs = 31, length = 2522883, total docs = 5507, last doc in bulk =
'http://blogotional.blogspot.com/2005_03_06_archive.html']
2015-10-13 16:47:43,778 INFO elastic.ElasticIndexWriter - Previous took in ms
827, including wait 766
2015-10-13 16:47:43,861 INFO elastic.ElasticIndexWriter - Processing bulk
request [docs = 29, length = 2570359, total docs = 5536, last doc in bulk =
'http://blogotional.blogspot.com/2005_09_25_archive.html']
2015-10-13 16:47:44,418 INFO elastic.ElasticIndexWriter - Previous took in ms
539, including wait 557
2015-10-13 16:47:44,512 INFO elastic.ElasticIndexWriter - Processing bulk
request [docs = 29, length = 2502887, total docs = 5565, last doc in bulk =
'http://blogotional.blogspot.com/2006_04_16_archive.html']
2015-10-13 16:47:45,348 INFO elastic.ElasticIndexWriter - Previous took in ms
855, including wait 836
2015-10-13 16:47:45,525 INFO elastic.ElasticIndexWriter - Processing bulk
request [docs = 158, length = 2515000, total docs = 5723, last doc in bulk =
'http://brazilcarroll.org/page/2/']
2015-10-13 16:47:45,969 INFO elastic.ElasticIndexWriter - Previous took in ms
552, including wait 444
2015-10-13 16:47:46,211 INFO elastic.ElasticIndexWriter - Processing bulk
request [docs = 250, length = 1141609, total docs = 5973, last doc in bulk =
'http://calvarybaptistwarren.com/page/trivia']
2015-10-13 16:47:47,058 INFO elastic.ElasticIndexWriter - Previous took in ms
1039, including wait 847
2015-10-13 16:47:47,288 INFO elastic.ElasticIndexWriter - Processing bulk
request [docs = 250, length = 1308537, total docs = 6223, last doc in bulk =
'http://catalystcommunitychurch.org/people/seth-barber/']
2015-10-13 16:47:47,398 INFO elastic.ElasticIndexWriter - Previous took in ms
302, including wait 110
2015-10-13 16:47:47,579 INFO elastic.ElasticIndexWriter - Processing bulk
request [docs = 122, length = 2500702, total docs = 6345, last doc in bulk =
'http://catholic-convert.com/resources/recommended/software/']
2015-10-13 16:47:48,057 INFO elastic.ElasticIndexWriter - Previous took in ms
611, including wait 478
2015-10-13 16:47:48,535 INFO elastic.ElasticIndexWriter - Processing bulk
request [docs = 250, length = 1727339, total docs = 6595, last doc in bulk =
'http://ccpville.com/2015/05/announcements-for-may-17-2015/']
2015-10-13 16:47:48,562 INFO elastic.ElasticIndexWriter - Previous took in ms
406, including wait 27
2015-10-13 16:47:48,878 INFO elastic.ElasticIndexWriter - Processing bulk
request [docs = 250, length = 1697119, total docs = 6845, last doc in bulk =
'http://cftministry.org/resources/bookmarks.html']
2015-10-13 16:47:49,356 INFO elastic.ElasticIndexWriter - Previous took in ms
747, including wait 478
2015-10-13 16:47:49,508 INFO elastic.ElasticIndexWriter - Processing bulk
request [docs = 250, length = 2183224, total docs = 7095, last doc in bulk =
'http://chicagoavenuechurchofchrist.org/new-years-resolution-christians/']
2015-10-13 16:47:49,771 INFO elastic.ElasticIndexWriter - Previous took in ms
315, including wait 263
2015-10-13 16:47:49,975 INFO elastic.ElasticIndexWriter - Processing bulk
request [docs = 250, length = 1717642, total docs = 7345, last doc in bulk =
'http://christevangelicalchurchmobmin.org/page/ministry_to_and_through_animals']
2015-10-13 16:47:50,455 INFO elastic.ElasticIndexWriter - Previous took in ms
599, including wait 480
2015-10-13 16:47:50,650 INFO elastic.ElasticIndexWriter - Processing bulk
request [docs = 250, length = 1600259, total docs = 7595, last doc in bulk =
'http://christianobserver.org/wp-includes/wlwmanifest.xml']
2015-10-13 16:47:51,021 INFO elastic.ElasticIndexWriter - Previous took in ms
507, including wait 371
2015-10-13 16:47:51,191 INFO elastic.ElasticIndexWriter - Processing bulk
request [docs = 144, length = 2548642, total docs = 7739, last doc in bulk =
'http://christlifedailybible.blogspot.com.au/2012_07_01_archive.html']
2015-10-13 16:47:51,525 INFO elastic.ElasticIndexWriter - Previous took in ms
460, including wait 334
2015-10-13 16:47:51,661 INFO elastic.ElasticIndexWriter - Processing bulk
request [docs = 156, length = 2501340, total docs = 7895, last doc in bulk =
'http://chuckanderson.blogspot.com/2013_01_01_archive.html']
2015-10-13 16:47:52,373 INFO elastic.ElasticIndexWriter - Previous took in ms
728, including wait 712
2015-10-13 16:47:52,645 INFO elastic.ElasticIndexWriter - Processing bulk
request [docs = 205, length = 3001522, total docs = 8100, last doc in bulk =
'http://classicalchristianity.com/category/bysaint/blessed-augustine-ca-354-430/']
2015-10-13 16:47:53,310 INFO elastic.ElasticIndexWriter - Previous took in ms
881, including wait 664
2015-10-13 16:47:53,392 INFO elastic.ElasticIndexWriter - Processing bulk
request [docs = 15, length = 2679457, total docs = 8115, last doc in bulk =
'http://classicalchristianity.com/category/bysaint/st-basil-of-caesarea-ca-330-379-%e3%80%80/']
2015-10-13 16:47:54,071 INFO elastic.ElasticIndexWriter - Previous took in ms
639, including wait 678
2015-10-13 16:47:54,157 INFO elastic.ElasticIndexWriter - Processing bulk
request [docs = 20, length = 2506012, total docs = 8135, last doc in bulk =
'http://classicalchristianity.com/category/canon-law/']
2015-10-13 16:47:54,970 INFO elastic.ElasticIndexWriter - Previous took in ms
750, including wait 813
2015-10-13 16:47:55,067 INFO elastic.ElasticIndexWriter - Processing bulk
request [docs = 13, length = 2964279, total docs = 8148, last doc in bulk =
'http://classicalchristianity.com/category/holyfathers/christology/']
2015-10-13 16:47:55,511 INFO elastic.ElasticIndexWriter - Previous took in ms
433, including wait 443
2015-10-13 16:47:55,598 INFO elastic.ElasticIndexWriter - Processing bulk
request [docs = 17, length = 2755989, total docs = 8165, last doc in bulk =
'http://classicalchristianity.com/category/sacrament/']
2015-10-13 16:47:56,298 INFO elastic.ElasticIndexWriter - Previous took in ms
722, including wait 699
2015-10-13 16:47:56,487 INFO elastic.ElasticIndexWriter - Processing bulk
request [docs = 107, length = 2598111, total docs = 8272, last doc in bulk =
'http://coffeehousebible.blogspot.com/2012_03_01_archive.html']
2015-10-13 16:47:56,781 INFO elastic.ElasticIndexWriter - Previous took in ms
384, including wait 294
2015-10-13 16:47:56,876 INFO elastic.ElasticIndexWriter - Processing bulk
request [docs = 46, length = 2540888, total docs = 8318, last doc in bulk =
'http://coffeehousebible.blogspot.com/2015_04_01_archive.html']
2015-10-13 16:47:57,608 INFO elastic.ElasticIndexWriter - Previous took in ms
766, including wait 732
2015-10-13 16:47:57,894 INFO elastic.ElasticIndexWriter - Processing bulk
request [docs = 221, length = 3106091, total docs = 8539, last doc in bulk =
'http://commfell.org/11009/ministry/ministry_id/301289/Men']
2015-10-13 16:47:58,091 INFO elastic.ElasticIndexWriter - Previous took in ms
359, including wait 197
2015-10-13 16:47:58,384 INFO elastic.ElasticIndexWriter - Processing bulk
request [docs = 250, length = 2241764, total docs = 8789, last doc in bulk =
'http://cornerstoneefree.org/mcintosh/']
2015-10-13 16:47:59,126 INFO elastic.ElasticIndexWriter - Previous took in ms
960, including wait 742
2015-10-13 16:47:59,374 INFO elastic.ElasticIndexWriter - Processing bulk
request [docs = 250, length = 2117950, total docs = 9039, last doc in bulk =
'http://crazycathie.ca/tag/quotes/']
2015-10-13 16:47:59,465 INFO elastic.ElasticIndexWriter - Previous took in ms
308, including wait 91
2015-10-13 16:47:59,760 INFO elastic.ElasticIndexWriter - Processing bulk
request [docs = 250, length = 2254727, total docs = 9289, last doc in bulk =
'http://cside.org/pastor-bryan-neal.aspx']
2015-10-13 16:47:59,970 INFO elastic.ElasticIndexWriter - Previous took in ms
458, including wait 209
2015-10-13 16:48:00,180 INFO elastic.ElasticIndexWriter - Processing bulk
request [docs = 250, length = 1911318, total docs = 9539, last doc in bulk =
'http://dailylightdevotional.org/01/0110.html']
2015-10-13 16:48:00,591 INFO elastic.ElasticIndexWriter - Previous took in ms
508, including wait 410
2015-10-13 16:48:00,746 INFO elastic.ElasticIndexWriter - Processing bulk
request [docs = 250, length = 1959251, total docs = 9789, last doc in bulk =
'http://davidmatthew.org.uk/wotwintro.html']
2015-10-13 16:48:01,061 INFO elastic.ElasticIndexWriter - Previous took in ms
433, including wait 315
2015-10-13 16:48:01,276 INFO elastic.ElasticIndexWriter - Processing bulk
request [docs = 250, length = 1000372, total docs = 10039, last doc in bulk =
'http://derekgriz.com/tag/student-ministry/']
2015-10-13 16:48:01,564 INFO elastic.ElasticIndexWriter - Previous took in ms
424, including wait 287
2015-10-13 16:48:01,795 INFO elastic.ElasticIndexWriter - Processing bulk
request [docs = 217, length = 2526991, total docs = 10256, last doc in bulk =
'http://diatheke.blogspot.com/2013_07_01_archive.html']
2015-10-13 16:48:01,883 INFO elastic.ElasticIndexWriter - Previous took in ms
255, including wait 88
2015-10-13 16:48:01,990 INFO elastic.ElasticIndexWriter - Processing bulk
request [docs = 47, length = 2545740, total docs = 10303, last doc in bulk =
'http://dictionaryofdoctrine.com/House-of-Cards.html']
2015-10-13 16:48:02,696 INFO elastic.ElasticIndexWriter - Previous took in ms
679, including wait 706
2015-10-13 16:48:02,894 INFO elastic.ElasticIndexWriter - Processing bulk
request [docs = 180, length = 2512128, total docs = 10483, last doc in bulk =
'http://distinctivediscipleship.com/category/daily-distinctives/']
2015-10-13 16:48:04,086 INFO elastic.ElasticIndexWriter - Previous took in ms
1271, including wait 1192
2015-10-13 16:48:04,222 INFO elastic.ElasticIndexWriter - Processing bulk
request [docs = 58, length = 2501110, total docs = 10541, last doc in bulk =
'http://doctrine.org/jesus-vs-paul/']
2015-10-13 16:48:05,479 INFO elastic.ElasticIndexWriter - Previous took in ms
1248, including wait 1257
2015-10-13 16:48:05,565 INFO elastic.ElasticIndexWriter - Processing bulk
request [docs = 42, length = 2520935, total docs = 10583, last doc in bulk =
'http://doctrine.org/understanding-the-book-of-revelation/']
2015-10-13 16:48:06,175 INFO elastic.ElasticIndexWriter - Previous took in ms
593, including wait 610
2015-10-13 16:48:06,406 INFO elastic.ElasticIndexWriter - Processing bulk
request [docs = 170, length = 2504080, total docs = 10753, last doc in bulk =
'http://doulogos.blogspot.com/2005/09/interview-what-happened.html']
2015-10-13 16:48:06,854 INFO elastic.ElasticIndexWriter - Previous took in ms
552, including wait 448
2015-10-13 16:48:06,939 INFO elastic.ElasticIndexWriter - Processing bulk
request [docs = 46, length = 2528772, total docs = 10799, last doc in bulk =
'http://doulogos.blogspot.com/2008_08_01_archive.html']
2015-10-13 16:48:07,511 INFO elastic.ElasticIndexWriter - Previous took in ms
603, including wait 572
2015-10-13 16:48:07,699 INFO elastic.ElasticIndexWriter - Processing bulk
request [docs = 141, length = 2505336, total docs = 10940, last doc in bulk =
'http://drmsh.com/category/archaeology/']
2015-10-13 16:48:08,146 INFO elastic.ElasticIndexWriter - Previous took in ms
567, including wait 447
2015-10-13 16:48:08,312 INFO elastic.ElasticIndexWriter - Processing bulk
request [docs = 194, length = 2557695, total docs = 11134, last doc in bulk =
'http://eaandfaith.blogspot.ca/2009_09_01_archive.html']
2015-10-13 16:48:08,758 INFO elastic.ElasticIndexWriter - Previous took in ms
551, including wait 446
2015-10-13 16:48:08,866 INFO elastic.ElasticIndexWriter - Processing bulk
request [docs = 78, length = 2573722, total docs = 11212, last doc in bulk =
'http://eaandfaith.blogspot.co.uk/2009_09_01_archive.html']
2015-10-13 16:48:09,361 INFO elastic.ElasticIndexWriter - Previous took in ms
534, including wait 495
2015-10-13 16:48:09,460 INFO elastic.ElasticIndexWriter - Processing bulk
request [docs = 78, length = 2574267, total docs = 11290, last doc in bulk =
'http://eaandfaith.blogspot.com.au/2009_09_01_archive.html']
2015-10-13 16:48:09,938 INFO elastic.ElasticIndexWriter - Previous took in ms
509, including wait 477
2015-10-13 16:48:10,042 INFO elastic.ElasticIndexWriter - Processing bulk
request [docs = 76, length = 2502863, total docs = 11366, last doc in bulk =
'http://eaandfaith.blogspot.com/2009_10_01_archive.html']
2015-10-13 16:48:10,343 INFO elastic.ElasticIndexWriter - Previous took in ms
328, including wait 300
2015-10-13 16:48:10,567 INFO elastic.ElasticIndexWriter - Processing bulk
request [docs = 204, length = 2548890, total docs = 11570, last doc in bulk =
'http://echoofrestorationtruths.blogspot.com/2013/10/the-language-of-beasts-of-revelation.html']
2015-10-13 16:48:10,894 INFO elastic.ElasticIndexWriter - Previous took in ms
447, including wait 327
2015-10-13 16:48:11,124 INFO elastic.ElasticIndexWriter - Processing bulk
request [docs = 250, length = 2158970, total docs = 11820, last doc in bulk =
'http://elbaptist.org/about/our-beliefs/civil-government']
2015-10-13 16:48:14,103 INFO elastic.ElasticIndexWriter - Previous took in ms
610, including wait 2979
2015-10-13 16:48:14,321 INFO elastic.ElasticIndexWriter - Processing bulk
request [docs = 239, length = 2543200, total docs = 12059, last doc in bulk =
'http://encountering-ahnsahnghong.blogspot.com/2012_02_01_archive.html']
2015-10-13 16:48:14,530 INFO elastic.ElasticIndexWriter - Previous took in ms
382, including wait 208
2015-10-13 16:48:14,682 INFO elastic.ElasticIndexWriter - Processing bulk
request [docs = 145, length = 2577069, total docs = 12204, last doc in bulk =
'http://endtimepilgrim.org/puritans12.htm']
2015-10-13 16:48:15,212 INFO elastic.ElasticIndexWriter - Previous took in ms
628, including wait 530
2015-10-13 16:48:15,356 INFO elastic.ElasticIndexWriter - Processing bulk
request [docs = 208, length = 2502224, total docs = 12412, last doc in bulk =
'http://english.genesis6.org/unmasking-sda-ellen-g-whites-satanic-hold-on-the-on-youtube/']
2015-10-13 16:48:23,467 INFO client.transport - [Grandmaster] failed to get
node info for
[#transport#-1][ci-dev-web06.lrscorp.net][inet[ci-dev-search04/10.70.15.17:9300]],
disconnecting...
org.elasticsearch.transport.ReceiveTimeoutTransportException:
[][inet[ci-dev-search04/10.70.15.17:9300]][cluster:monitor/nodes/info]
request_id [101] timed out after [5002ms]
at
org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:366)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2015-10-13 16:48:33,475 INFO client.transport - [Grandmaster] failed to get
node info for
[#transport#-1][ci-dev-web06.lrscorp.net][inet[ci-dev-search04/10.70.15.17:9300]],
disconnecting...
org.elasticsearch.transport.ReceiveTimeoutTransportException:
[][inet[ci-dev-search04/10.70.15.17:9300]][cluster:monitor/nodes/info]
request_id [102] timed out after [5000ms]
at
org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:366)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2015-10-13 16:48:37,246 INFO elastic.ElasticIndexWriter - Previous took in ms
21903, including wait 21890
2015-10-13 16:48:37,247 INFO elastic.ElasticIndexWriter - Processing remaining
requests [docs = 208, length = 2502224, total docs = 12412]
2015-10-13 16:48:37,255 WARN mapred.LocalJobRunner - job_local1050818242_0001
org.elasticsearch.client.transport.NoNodeAvailableException: None of the
configured nodes are available: []
at
org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:278)
at
org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:197)
at
org.elasticsearch.client.transport.support.InternalTransportClient.execute(InternalTransportClient.java:106)
at
org.elasticsearch.client.support.AbstractClient.bulk(AbstractClient.java:163)
at
org.elasticsearch.client.transport.TransportClient.bulk(TransportClient.java:364)
at
org.elasticsearch.action.bulk.BulkRequestBuilder.doExecute(BulkRequestBuilder.java:164)
at
org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:91)
at
org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:65)
at
org.apache.nutch.indexwriter.elastic.ElasticIndexWriter.commit(ElasticIndexWriter.java:211)
at
org.apache.nutch.indexwriter.elastic.ElasticIndexWriter.write(ElasticIndexWriter.java:161)
at org.apache.nutch.indexer.IndexWriters.write(IndexWriters.java:85)
at
org.apache.nutch.indexer.IndexerOutputFormat$1.write(IndexerOutputFormat.java:50)
at
org.apache.nutch.indexer.IndexerOutputFormat$1.write(IndexerOutputFormat.java:41)
at
org.apache.hadoop.mapred.ReduceTask$OldTrackingRecordWriter.write(ReduceTask.java:458)
at org.apache.hadoop.mapred.ReduceTask$3.collect(ReduceTask.java:500)
at
org.apache.nutch.indexer.IndexerMapReduce.reduce(IndexerMapReduce.java:337)
at
org.apache.nutch.indexer.IndexerMapReduce.reduce(IndexerMapReduce.java:53)
at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:522)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:421)
at
org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:398)
2015-10-13 16:48:37,542 ERROR indexer.IndexingJob - Indexer:
java.io.IOException: Job failed!
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1357)
at org.apache.nutch.indexer.IndexingJob.index(IndexingJob.java:113)
at org.apache.nutch.indexer.IndexingJob.run(IndexingJob.java:177)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.nutch.indexer.IndexingJob.main(IndexingJob.java:187)
Here's my elastic search log for the same timeframe:
[2015-10-13 01:14:07,445][INFO ][cluster.service ] [ci-dev-search04]
removed {[La
Lunatica][e0LVxHLbSKGYNhEavjIJXw][ws1938][inet[ws1890.lrscorp.net/10.200.208.38:9300]]{data=false,
client=true},}, reason: zen-disco-node_failed([La
Lunatica][e0LVxHLbSKGYNhEavjIJXw][ws1938][inet[ws1890.lrscorp.net/10.200.208.38:9300]]{data=false,
client=true}), reason transport disconnected
[2015-10-13 16:18:09,749][WARN ][monitor.jvm ] [ci-dev-search04]
[gc][young][54349][4966] duration [23.3s], collections [4]/[24.1s], total
[23.3s]/[50.8s], memory [205.2mb]->[177.9mb]/[989.8mb], all_pools {[young]
[56.9mb]->[112.6kb]/[273mb]}{[survivor] [8.5mb]->[8.5mb]/[34.1mb]}{[old]
[139.7mb]->[169.3mb]/[682.6mb]}
[2015-10-13 16:23:09,914][WARN ][monitor.jvm ] [ci-dev-search04]
[gc][young][54629][5066] duration [20.1s], collections [1]/[20.4s], total
[20.1s]/[1.2m], memory [171.5mb]->[153.4mb]/[989.8mb], all_pools {[young]
[25.2mb]->[1.7mb]/[273mb]}{[survivor] [4.1mb]->[8.5mb]/[34.1mb]}{[old]
[142.1mb]->[143.5mb]/[682.6mb]}
[2015-10-13 16:47:13,468][WARN ][monitor.jvm ] [ci-dev-search04]
[gc][young][56067][5203] duration [5s], collections [1]/[5.1s], total
[5s]/[1.3m], memory [248.1mb]->[191.2mb]/[989.8mb], all_pools {[young]
[66mb]->[894.5kb]/[273mb]}{[survivor] [8.5mb]->[8.5mb]/[34.1mb]}{[old]
[173.6mb]->[181.9mb]/[682.6mb]}
[2015-10-13 16:47:23,360][WARN ][monitor.jvm ] [ci-dev-search04]
[gc][young][56071][5213] duration [6.3s], collections [2]/[6.8s], total
[6.3s]/[1.4m], memory [267.5mb]->[279.2mb]/[989.8mb], all_pools {[young]
[196.7kb]->[165.7kb]/[273mb]}{[survivor] [8.5mb]->[7.8mb]/[34.1mb]}{[old]
[258.9mb]->[271.2mb]/[682.6mb]}
[2015-10-13 16:48:13,461][WARN ][monitor.jvm ] [ci-dev-search04]
[gc][young][56119][5353] duration [2.4s], collections [1]/[2.7s], total
[2.4s]/[1.5m], memory [296.7mb]->[257.5mb]/[989.8mb], all_pools {[young]
[48.2mb]->[785.7kb]/[273mb]}{[survivor] [8.5mb]->[8.5mb]/[34.1mb]}{[old]
[239.9mb]->[248.4mb]/[682.6mb]}
[2015-10-13 16:48:36,621][WARN ][monitor.jvm ] [ci-dev-search04]
[gc][young][56122][5360] duration [21s], collections [1]/[21.1s], total
[21s]/[1.9m], memory [334.2mb]->[311.6mb]/[989.8mb], all_pools {[young]
[29.1mb]->[991.5kb]/[273mb]}{[survivor] [3.1mb]->[8.5mb]/[34.1mb]}{[old]
[302mb]->[302.3mb]/[682.6mb]}
