mbien opened a new pull request, #5922:
URL: https://github.com/apache/netbeans/pull/5922
wip
- index only interesting fields to reduce index size
- this is done by using a custom index creator since filters can't remove
fields
- reduces central index size from 5.7GB to 2.3GB
- todo: artifact date cutoff filter needs UI
- more robust behavior when an incompatible index is encountered
- clears the index folder
The filtering is implemented via a custom index creator, for inexplicable
reasons I couldn't reproduce my results from a few month ago where i removed
fields in the extraction filter. Removing fields there doesn't do anything so
we do that one step earlier in the pipeline.
benchmarks (I tested on JDK 19 since the current lucene version won't use
panama on JDK 20+, next update will fix this):
```
# sh1, desc+info filter, full index
762.92 s
2,3 GB
# MT enabled
328.40 s
3,3 GB
index size after first "lucene gc":
2,8 GB
```
```
# sh1, desc+info filter, 2y index
466.80 s
1,1 GB
# MT enabled
264.34 s
1,3 GB
```
reference:
```
# no filters, full index:
916.25 s
5.7 GB
```
note: time does not include the download since for testing purposes I load
the gz from disk.
note2: MT causes index size inefficiencies. However, some of it is "garbage
collected" by lucene automatically when the index is used a few times. (search
a few artifacts + restart NB)
note3: this is an old i6700k, storage is NVMe.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
