| Yurik added a comment. |
I just finished full re-import with -Xmx=16GB , and it worked fine. I am not sure what has caused the original issue. For future reference, here are the import stats (importing both Wikidata and OSM data from the same dir - 33GB total, 683 files of about the same size. Total time was about 36 hours. First 329 files were OSM, second 354 - wikidata (note how the graph starts jumping, possibly due to per-file statement distribution in WD. OSM data is much more uniform in terms of statement per osm object)
raw data
statements time rate sets 107,179,984 640 167,220 6 189,557,036 1,260 150,378 11 255,343,483 1,884 135,523 15 323,230,658 2,533 127,587 19 389,362,380 3,231 120,504 23 454,181,776 3,971 114,351 27 519,770,290 4,654 111,667 31 585,537,711 5,358 109,275 35 669,713,329 6,025 111,138 40 753,727,112 6,704 112,415 45 835,803,507 7,485 111,656 50 901,307,143 8,218 109,668 54 966,655,528 8,963 107,845 58 1,015,648,545 9,570 106,123 61 1,080,889,536 10,310 104,829 65 1,147,921,605 11,098 103,427 69 1,218,376,248 11,800 103,245 73 1,288,894,078 12,464 103,401 77 1,340,627,982 13,252 101,158 80 1,374,208,625 13,948 98,521 82 1,408,174,205 14,568 96,659 84 1,459,277,475 15,389 94,824 87 1,510,100,090 16,207 93,174 90 1,560,517,757 16,986 91,866 93 1,627,435,035 17,713 91,877 97 1,693,942,260 18,401 92,053 101 1,760,534,757 19,151 91,927 105 1,827,353,450 19,934 91,669 109 1,877,349,669 20,571 91,260 112 1,927,029,732 21,190 90,937 115 1,976,839,801 21,803 90,665 118 2,043,621,669 22,602 90,417 122 2,093,614,307 23,222 90,153 125 2,143,530,955 23,884 89,744 128 2,193,300,587 24,504 89,506 131 2,259,768,850 25,281 89,382 135 2,309,402,067 25,896 89,178 138 2,359,045,834 26,524 88,937 141 2,408,813,042 27,193 88,582 144 2,458,534,137 27,866 88,224 147 2,508,173,248 28,587 87,736 150 2,557,695,348 29,236 87,481 153 2,607,093,365 29,879 87,254 156 2,672,980,080 30,659 87,181 160 2,722,486,059 31,293 86,997 163 2,771,962,432 31,913 86,859 166 2,837,660,998 32,616 87,001 170 2,887,208,486 33,270 86,779 173 2,936,799,169 33,925 86,567 176 2,986,273,795 34,593 86,325 179 3,035,728,568 35,243 86,136 182 3,085,184,315 35,859 86,035 185 3,151,468,039 36,608 86,086 189 3,201,011,063 37,213 86,017 192 3,267,535,396 37,935 86,134 196 3,334,686,516 38,576 86,443 200 3,401,886,904 39,240 86,692 204 3,468,645,120 39,954 86,814 208 3,534,648,676 40,730 86,781 212 3,600,523,802 41,492 86,775 216 3,666,138,573 42,282 86,706 220 3,731,829,153 43,077 86,629 224 3,797,535,576 43,829 86,643 228 3,863,523,948 44,582 86,659 232 3,928,985,444 45,258 86,812 236 3,994,209,028 45,921 86,979 240 4,059,507,043 46,604 87,104 244 4,125,187,426 47,359 87,104 248 4,190,652,309 48,108 87,108 252 4,256,123,933 48,782 87,246 256 4,321,507,795 49,466 87,362 260 4,386,987,648 50,120 87,528 264 4,452,442,948 50,739 87,750 268 4,517,769,637 51,363 87,956 272 4,583,567,763 52,006 88,133 276 4,649,357,008 52,705 88,213 280 4,715,013,078 53,387 88,316 284 4,780,538,748 54,036 88,468 288 4,845,958,188 54,701 88,588 292 4,912,126,601 55,324 88,787 296 4,977,535,397 55,951 88,961 300 5,042,533,501 56,599 89,091 304 5,107,487,478 57,217 89,264 308 5,172,445,286 57,824 89,450 312 5,253,524,269 58,545 89,734 317 5,305,784,367 59,194 89,633 320 5,343,676,632 60,030 89,016 322 5,381,274,598 60,966 88,265 324 5,418,420,058 61,807 87,665 326 5,468,584,289 62,692 87,228 329 5,518,098,321 63,442 86,977 333 5,564,009,469 64,131 86,759 339 5,619,110,296 64,763 86,763 345 5,673,559,079 65,380 86,777 354 5,715,132,031 65,986 86,610 360 5,756,082,773 66,616 86,406 368 5,812,130,630 67,248 86,428 376 5,858,211,979 67,888 86,291 382 5,917,602,276 68,563 86,308 390 5,972,627,425 69,255 86,241 396 6,033,689,471 70,387 85,720 400 6,061,126,737 71,089 85,260 402 6,090,631,401 71,841 84,778 405 6,120,734,012 72,539 84,377 409 6,153,355,223 73,182 84,082 413 6,207,067,286 73,873 84,023 420 6,253,755,885 74,611 83,817 427 6,284,713,321 75,280 83,484 432 6,319,282,922 76,000 83,147 439 6,355,867,878 76,609 82,964 444 6,399,714,387 77,288 82,803 450 6,436,640,839 78,036 82,482 455 6,481,234,379 78,650 82,405 461 6,526,759,664 79,391 82,209 466 6,579,995,114 80,501 81,737 470 6,599,378,396 81,211 81,261 471 6,625,131,210 82,089 80,705 473 6,645,098,333 82,772 80,281 475 6,669,624,733 83,502 79,873 478 6,699,812,271 84,314 79,462 482 6,738,122,743 84,990 79,281 486 6,774,046,102 85,662 79,078 492 6,808,866,336 86,267 78,927 497 6,833,736,316 86,915 78,625 501 6,860,334,945 87,592 78,320 506 6,887,971,112 88,315 77,992 511 6,919,244,750 88,952 77,786 515 6,953,331,447 89,571 77,628 520 6,978,517,812 90,206 77,361 523 7,007,112,877 90,843 77,134 527 7,044,596,712 91,468 77,016 532 7,080,015,049 92,212 76,779 536 7,124,088,113 93,041 76,569 540 7,145,384,434 94,007 76,008 541 7,163,582,483 94,892 75,491 542 7,177,490,061 95,537 75,127 543 7,198,819,202 96,483 74,611 545 7,217,160,866 97,276 74,191 547 7,239,559,213 98,115 73,786 550 7,263,224,958 98,910 73,432 553 7,291,006,386 99,618 73,189 556 7,327,155,852 100,322 73,035 561 7,348,363,609 100,952 72,790 565 7,379,345,399 101,720 72,545 569 7,396,569,140 102,359 72,260 572 7,417,391,783 103,081 71,956 576 7,438,740,940 103,845 71,632 580 7,467,690,245 104,563 71,417 584 7,488,657,687 105,164 71,208 587 7,518,220,972 105,879 71,007 591 7,540,436,150 106,623 70,719 594 7,562,375,588 107,282 70,490 597 7,598,002,915 108,023 70,336 602 7,625,559,187 108,697 70,154 605 7,653,098,007 109,323 70,004 608 7,687,155,971 110,042 69,856 611 7,710,473,517 111,457 69,178 612 7,727,861,201 112,465 68,713 613 7,740,780,981 113,152 68,410 614 7,761,818,411 114,232 67,947 616 7,779,652,176 115,128 67,573 618 7,794,807,466 115,763 67,333 620 7,808,565,243 116,378 67,096 622 7,826,326,493 117,025 66,876 624 7,842,746,787 117,642 66,666 626 7,877,394,670 118,251 66,615 630 7,894,298,591 118,872 66,409 633 7,918,025,733 119,505 66,256 637 7,941,220,171 120,317 66,002 640 7,958,978,075 121,044 65,752 643 7,974,745,220 121,677 65,539 646 7,994,758,935 122,487 65,270 650 8,013,719,634 123,209 65,041 653 8,037,431,063 123,928 64,855 656 8,063,684,080 124,612 64,709 660 8,088,059,972 125,510 64,441 663 8,110,100,469 126,283 64,221 666 8,130,658,259 126,996 64,022 669 8,160,908,753 127,709 63,901 673 8,188,651,027 128,478 63,735 676 8,215,993,683 129,194 63,594 679 8,250,187,727 129,996 63,464 682
TASK DETAIL
EMAIL PREFERENCES
To: Yurik
Cc: Gehel, Smalyshev, Aklapper, Yurik, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, Avner, debt, Jonas, FloNight, Xmlizer, Izno, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
Cc: Gehel, Smalyshev, Aklapper, Yurik, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, Avner, debt, Jonas, FloNight, Xmlizer, Izno, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
_______________________________________________ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
