[
https://issues.apache.org/jira/browse/LUCENE-7306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adrien Grand updated LUCENE-7306:
---------------------------------
Attachment: LUCENE-7306.patch
I switched to radix sort for other dimensions and HeapPointsWriter too and the
speedup is quite spectacular. This time it only took 241s to index (-30%). Both
flushing and merging are significantly faster:
Master flush
{noformat}
IW 0 [2016-05-26T16:28:30.643Z; Thread-0]: 261 msec to write points
IW 0 [2016-05-26T16:28:30.987Z; Thread-0]: 176 msec to write points
IW 0 [2016-05-26T16:28:31.238Z; Thread-0]: 172 msec to write points
IW 0 [2016-05-26T16:28:31.457Z; Thread-0]: 146 msec to write points
IW 0 [2016-05-26T16:28:31.676Z; Thread-0]: 147 msec to write points
IW 0 [2016-05-26T16:28:31.894Z; Thread-0]: 145 msec to write points
IW 0 [2016-05-26T16:28:32.114Z; Thread-0]: 146 msec to write points
IW 0 [2016-05-26T16:28:32.330Z; Thread-0]: 143 msec to write points
IW 0 [2016-05-26T16:28:32.549Z; Thread-0]: 146 msec to write points
IW 0 [2016-05-26T16:28:32.774Z; Thread-0]: 147 msec to write points
IW 0 [2016-05-26T16:28:35.450Z; Thread-0]: 157 msec to write points
IW 0 [2016-05-26T16:28:35.704Z; Thread-0]: 150 msec to write points
IW 0 [2016-05-26T16:28:35.925Z; Thread-0]: 152 msec to write points
IW 0 [2016-05-26T16:28:36.141Z; Thread-0]: 148 msec to write points
IW 0 [2016-05-26T16:28:36.357Z; Thread-0]: 149 msec to write points
{noformat}
Patch flush
{noformat}
IW 0 [2016-05-27T10:48:42.854Z; Thread-0]: 213 msec to write points
IW 0 [2016-05-27T10:48:43.117Z; Thread-0]: 138 msec to write points
IW 0 [2016-05-27T10:48:43.343Z; Thread-0]: 108 msec to write points
IW 0 [2016-05-27T10:48:43.518Z; Thread-0]: 91 msec to write points
IW 0 [2016-05-27T10:48:43.686Z; Thread-0]: 97 msec to write points
IW 0 [2016-05-27T10:48:43.856Z; Thread-0]: 99 msec to write points
IW 0 [2016-05-27T10:48:44.039Z; Thread-0]: 100 msec to write points
IW 0 [2016-05-27T10:48:44.231Z; Thread-0]: 100 msec to write points
IW 0 [2016-05-27T10:48:44.415Z; Thread-0]: 97 msec to write points
IW 0 [2016-05-27T10:48:44.603Z; Thread-0]: 96 msec to write points
IW 0 [2016-05-27T10:48:47.069Z; Thread-0]: 107 msec to write points
IW 0 [2016-05-27T10:48:47.230Z; Thread-0]: 92 msec to write points
IW 0 [2016-05-27T10:48:47.391Z; Thread-0]: 92 msec to write points
IW 0 [2016-05-27T10:48:47.550Z; Thread-0]: 92 msec to write points
IW 0 [2016-05-27T10:48:47.711Z; Thread-0]: 93 msec to write points
{noformat}
Master merge
{noformat}
SM 0 [2016-05-26T16:28:35.224Z; Thread-0]: 2414 msec to merge points [1096300
docs]
SM 0 [2016-05-26T16:28:39.390Z; Thread-0]: 1899 msec to merge points [1096300
docs]
SM 0 [2016-05-26T16:28:43.443Z; Thread-0]: 1869 msec to merge points [1096300
docs]
SM 0 [2016-05-26T16:28:47.426Z; Thread-0]: 1812 msec to merge points [1096300
docs]
SM 0 [2016-05-26T16:28:51.444Z; Thread-0]: 1850 msec to merge points [1096300
docs]
SM 0 [2016-05-26T16:28:55.422Z; Thread-0]: 1819 msec to merge points [1096300
docs]
SM 0 [2016-05-26T16:28:59.409Z; Thread-0]: 1823 msec to merge points [1096300
docs]
SM 0 [2016-05-26T16:29:03.368Z; Thread-0]: 1817 msec to merge points [1096300
docs]
SM 0 [2016-05-26T16:29:07.296Z; Thread-0]: 1802 msec to merge points [1096300
docs]
SM 0 [2016-05-26T16:29:11.205Z; Thread-0]: 1793 msec to merge points [1096300
docs]
SM 0 [2016-05-26T16:29:34.980Z; Thread-0]: 23722 msec to merge points [10963000
docs]
SM 0 [2016-05-26T16:29:38.934Z; Thread-0]: 1798 msec to merge points [1096300
docs]
SM 0 [2016-05-26T16:29:42.844Z; Thread-0]: 1779 msec to merge points [1096300
docs]
SM 0 [2016-05-26T16:29:46.849Z; Thread-0]: 1797 msec to merge points [1096300
docs]
SM 0 [2016-05-26T16:29:50.866Z; Thread-0]: 1802 msec to merge points [1096300
docs]
{noformat}
Patch merge
{noformat}
SM 0 [2016-05-27T10:48:46.890Z; Thread-0]: 2252 msec to merge points [1096300
docs]
SM 0 [2016-05-27T10:48:49.943Z; Thread-0]: 1184 msec to merge points [1096300
docs]
SM 0 [2016-05-27T10:48:52.722Z; Thread-0]: 1157 msec to merge points [1096300
docs]
SM 0 [2016-05-27T10:48:55.440Z; Thread-0]: 1140 msec to merge points [1096300
docs]
SM 0 [2016-05-27T10:48:58.146Z; Thread-0]: 1169 msec to merge points [1096300
docs]
SM 0 [2016-05-27T10:49:00.915Z; Thread-0]: 1200 msec to merge points [1096300
docs]
SM 0 [2016-05-27T10:49:03.684Z; Thread-0]: 1149 msec to merge points [1096300
docs]
SM 0 [2016-05-27T10:49:06.436Z; Thread-0]: 1126 msec to merge points [1096300
docs]
SM 0 [2016-05-27T10:49:09.169Z; Thread-0]: 1148 msec to merge points [1096300
docs]
SM 0 [2016-05-27T10:49:11.903Z; Thread-0]: 1165 msec to merge points [1096300
docs]
SM 0 [2016-05-27T10:49:28.140Z; Thread-0]: 16179 msec to merge points [10963000
docs]
SM 0 [2016-05-27T10:49:30.913Z; Thread-0]: 1159 msec to merge points [1096300
docs]
SM 0 [2016-05-27T10:49:33.609Z; Thread-0]: 1135 msec to merge points [1096300
docs]
SM 0 [2016-05-27T10:49:36.383Z; Thread-0]: 1149 msec to merge points [1096300
docs]
SM 0 [2016-05-27T10:49:39.118Z; Thread-0]: 1143 msec to merge points [1096300
docs]
{noformat}
> Use radix sort for points too
> -----------------------------
>
> Key: LUCENE-7306
> URL: https://issues.apache.org/jira/browse/LUCENE-7306
> Project: Lucene - Core
> Issue Type: Improvement
> Reporter: Adrien Grand
> Assignee: Adrien Grand
> Priority: Minor
> Attachments: LUCENE-7306.patch, LUCENE-7903.patch
>
>
> Like postings, points make heavy use of sorting at indexing time, so we
> should try to leverage radix sort too?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
