rzo1 opened a new pull request, #568:
URL: https://github.com/apache/opennlp/pull/568
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [x] Does your PR title start with OPENNLP-XXXX where XXXX is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [x] Has your PR been rebased against the latest commit within the target
branch (typically main)?
- [ ] Is your initial contribution a single, squashed commit?
### For code changes:
- [x] Have you ensured that the full suite of tests is executed via mvn
clean install at the root opennlp folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [x] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the LICENSE file, including the main
LICENSE file in opennlp folder?
- [ ] If applicable, have you updated the NOTICE file, including the main
NOTICE file found in opennlp folder?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which
it is rendered?
### Note:
I found
[OPENNLP-421](https://issues.apache.org/jira/projects/OPENNLP/issues/OPENNLP-421)
in our issue tracker and added a simple JMH benchmark to look into the impact
of interning strings and our multiple copy-operations in `StringList`.
- Here is a JMH Benchmark for the vanilla version of OpenNLP:
https://gist.github.com/rzo1/d2ba3e48c6bc190977baf9ee42388823
- Here is a JMH Benchmark with String interning removed from `StringList`
and usage of `System.arraycopy(...)`:
https://gist.github.com/rzo1/327c84ebac18b62baf927f1a87ec7480
As stated by the original issue opener, String interning was most likely
used because of:
> Presumably this is an attempt to reduce memory usage for duplicate tokens.
Interned Strings are stored in the JVM's permanent generation, which has a
small fixed size (seems to be about 83 MB on modern 64-bit JVMs:
https://www.oracle.com/java/technologies/javase/vmoptions-jsp.html)
So if we have huge `StringLists` or `Dictionaries`, users need to increase
`-XX:MaxPermSize=` option to avoid an OutOfMemoryException.
There might also be room for improvement in the `Dictionary` implemention as
we mess around with `StringLists`, `StringListWrappers`, etc. - but that would
be something to look into next.
Just want to have some thoughts on the interning removal here ;-)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@opennlp.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
