This is an automated email from the ASF dual-hosted git repository.
krickert pushed a change to branch OPENNLP-1982-embed-padding
in repository https://gitbox.apache.org/repos/asf/opennlp.git
omit 89205f1e6 Add padding strategies for embedAll
omit 4cf8fb31a Cap the token positions of one embedAll inference
omit 3d7395de4 SentenceVectorsDL cannot run on a GPU
omit 756e58ee8 GPU inference fails on the first CUDA request in a fresh JVM
add 47369544c (fix) GH-1378: MeCab CSV reader accepts unterminated quoted
fields (#1383)
add a368de9cd (fix) GH-1379: Morpheme accepts a surface that does not
match its span (#1384)
add 197933cf9 OPENNLP-1934: Add explicit tokenizer character policies
(#1349)
add 42877392f Minor: Use ArgumentChecks for inline null checks in
normalizer and tokenize (#1400)
add fce5fdbd6 Minor: Use ArgumentChecks for the inline null checks in
opennlp-api (#1399)
add c8536d2c1 Minor: Use ArgumentChecks for inline null checks in ml,
model-resolver, spellcheck (#1402)
add 35e97920d Minor: Use ArgumentChecks for the remaining inline null
checks in runtime (#1401)
add d742b18f0 MorfologikTagDictionary is not thread-safe and fails on
entries without a tag (#1382)
add b2e843340 Bump org.apache.maven:apache-maven from 3.9.16 to 3.10.0
(#1405)
add dd45c0042 Bump com.puppycrawl.tools:checkstyle from 13.3.0 to 14.3.0
(#1407)
add a6871186b GH-1408: Rename ArgumentChecks to ParamChecks (#1409)
add 6c8fdd5c7 GH-1397: Reject null spellcheck arguments with
IllegalArgumentException (#1412)
add 169827669 GH-1396: Use ParamChecks for null arguments in
opennlp-runtime (#1411)
add 66b1719bd GH-1395: Reject null arguments in opennlp-api with
IllegalArgumentException (#1410)
add 19f201e80 GH-1414: Store mvnw.cmd with LF line endings (#1415)
add c98b76298 GH-1398: Add requireNonEmpty and requireNonNegative to
ParamChecks (#1413)
add 1fd51808e Spellcheck correction loses the capital of a
supplementary-plane initial (#1385)
add 8b8a4dc86 GH-1422: Clean up opennlp-morfologik builder and Javadoc
(#1430)
add af742d8bd GH-1422: Clean up NumberUtil and its tests in opennlp-uima
(#1432)
add e1a807c47 GH-1388: Load bundled Unicode data through one recoverable
loader (#1373)
add 42f446a5a GH-1422: Clean up the spellcheck normalizer and loader
(#1431)
add 95b1ac5ff GH-1422: Clean up the report listeners in opennlp-cli (#1429)
add d7eadeda2 GH-1422: Clean up the stemmer, Hunspell and NameFinderDL
evals (#1434)
add 0a363d573 GH-1421: Clean up the AD, MASC, CoNLL-U and Brat readers
(#1428)
add 97e4fb13f GH-1418: Clean up event parsing and context generators in
the ml modules (#1425)
add 0ca246230 GH-1417: Remove duplicated logic in opennlp-api and the
lattice tokenizer (#1424)
add ef04afe6d GH-1416: Correct and trim opennlp-api Javadoc (#1423)
add 9a6a927a5 GH-1422: Remove duplication in StringUtilTest and the
chunker listener test (#1433)
add 11a136cc6 GH-1441: Report malformed lattice dictionary content as
InvalidFormatException (#1443)
add bca31bb7c OPENNLP-1988: Load WordPiece vocabularies from
tokenizer.json (#1319)
add 013b789b1 GH-1394: Load downloaded models by language when the model
host cannot be reached (#1435)
add 5a6347ac2 GH-1436: Document thread-safety limits of
WhitespaceTokenizer.INSTANCE and the process-wide modes (#1437)
add cf81387cc GH-1438: Share code point validation and trim scans in
CharClass and CodePointSet (#1439)
add 617d38996 GH-1440: Isolate WordpieceTokenizer from the shared
WhitespaceTokenizer and share punctuation isolation (#1442)
add a9d226ce9 GH-1445: Run the CI build once per job and render
README.html from README.md only (#1446)
add e039a7fb7 GH-1393: Cache the downloaded test models between CI runs
(#1444)
add deb80bae8 OPENNLP-547: Add a transition-based dependency parser
component (#1236)
add 139b9ddb3 GH-1449: Share ONNX inference between DocumentCategorizerDL
and NameFinderDL with one failure message (#1450)
add ce2da5357 GH-1447: Reject self-closing AD markup tags as opening tags
(#1448)
add 779420863 GH-1451: Share staged install, checksum and tar helpers
across the dictionary installers (#1452)
add 8d2a8c310 GH-1419: Correct and trim opennlp-runtime Javadoc (#1426)
add eef3909dc GH-1420: Remove duplication and add readResolve in
opennlp-runtime (#1427)
add ef814bdba GH-1454: Tidy document, stemmer, corpora and doccat chapters
and README (#1455)
add 84a1d6268 GH-1460: Match escaped tags in the bundled Spanish head
rules (#1461)
add e286928a1 GH-1458: Parse name-finder tags and parse-tree strings
without regular expressions (#1459)
add a6f19f91e GH-1449: Align DocumentCategorizerDLEval with the shared
ONNX failure message (#1463)
add 46d9d96a1 GH-1456: Tighten normalizer, tokenizer, introduction and
language detector chapters (#1457)
add 7a560bd7a OPENNLP-1935: Regex removal (10/10): Forbid java.util.regex
in production code (#1348)
add 576206d29 GH-1389, GH-1390: Let SentenceVectorsDL run on a GPU and fix
the first CUDA request in a fresh JVM
add 99d88ba09 GH-1389: Skip the opennlp-dl-gpu tests where onnxruntime_gpu
has no native library
add 8d4e792d1 GH-1389: Share the tiny model test fixtures
add bcdd68d90 GH-1391: Test that embedAll caps the token positions of one
inference
add cd2fbba3c GH-1391: Cap the token positions of one embedAll inference
add 6557e4c4d GH-1391: Shorten the embedAll Javadoc and compute the
expected batch sizes
add 8592377fb GH-1391: Test the batch boundaries and split groups of
embedAll
add 82156b04f GH-1391: Share the batch test fixtures and drop duplicate
tests
add 8c1bc0759 GH-1392: Add padding strategies for embedAll
add 8c7c439df GH-1392: Pad to the longest text by default on a GPU
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (89205f1e6)
\
N -- N -- N refs/heads/OPENNLP-1982-embed-padding (8c7c439df)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.github/CONTRIBUTING.md | 14 +
.github/workflows/maven.yml | 19 +-
.mvn/wrapper/maven-wrapper.properties | 2 +-
README.md | 10 +-
dev/README-hunspell-dictionaries.md | 4 +-
dev/forbidden-regex.txt | 30 ++
.../opennlp/tools/commons/SuppressForbidden.java | 51 ++
.../java/opennlp/tools/depparse/DependencyArc.java | 65 +++
.../DependencyEvaluationMonitor.java} | 8 +-
.../opennlp/tools/depparse/DependencyParser.java | 50 ++
.../opennlp/tools/depparse/DependencySample.java | 145 ++++++
.../opennlp/tools/depparse/DependencyTree.java | 270 +++++++++++
.../java/opennlp/tools/doccat/DocumentSample.java | 7 +-
.../java/opennlp/tools/document/Annotation.java | 5 +-
.../main/java/opennlp/tools/document/Document.java | 42 +-
.../opennlp/tools/document/DocumentAnalyzer.java | 6 +-
.../opennlp/tools/document/DocumentAnnotator.java | 5 +-
.../opennlp/tools/document/DocumentAnnotators.java | 27 +-
.../opennlp/tools/document/DocumentMerges.java | 71 +++
.../opennlp/tools/document/ImmutableDocument.java | 41 +-
.../main/java/opennlp/tools/document/LayerKey.java | 5 +-
.../main/java/opennlp/tools/document/Layers.java | 24 +-
.../opennlp/tools/embeddings/TextEmbedder.java | 6 +-
.../java/opennlp/tools/langdetect/Language.java | 6 +-
.../opennlp/tools/langdetect/LanguageSample.java | 15 +-
.../main/java/opennlp/tools/ml/model/Event.java | 13 +-
.../java/opennlp/tools/namefind/NameSample.java | 56 ++-
.../tools/namefind/OffsetMappingNameFinder.java | 13 +-
.../src/main/java/opennlp/tools/parser/Parse.java | 114 +++--
.../opennlp/tools/sentiment/SentimentSample.java | 9 +-
.../java/opennlp/tools/stemmer/StemmerFactory.java | 4 +-
.../java/opennlp/tools/termvector/TermVector.java | 13 +-
.../opennlp/tools/tokenize/BertNormalization.java | 49 +-
.../java/opennlp/tools/tokenize/MappedText.java | 161 +++++++
.../java/opennlp/tools/tokenize/SubwordPiece.java | 14 +-
.../opennlp/tools/tokenize/SubwordTokenizer.java | 5 +
.../java/opennlp/tools/tokenize/TokenSample.java | 19 +-
.../tools/tokenize/WhitespaceTokenizer.java | 8 +-
.../opennlp/tools/tokenize/WordpieceEncoder.java | 195 ++------
.../opennlp/tools/tokenize/WordpieceTokenizer.java | 62 +--
.../java/opennlp/tools/util/ArgumentChecks.java | 57 ---
.../java/opennlp/tools/util/CompatibilityMode.java | 24 +-
.../opennlp/tools/util/LanguageCodeValidator.java | 13 +-
.../java/opennlp/tools/util/ObjectStreamUtils.java | 12 +-
.../main/java/opennlp/tools/util/ParamChecks.java | 223 +++++++++
.../opennlp/tools/util/PlainTextByLineStream.java | 7 +-
.../java/opennlp/tools/util/ResourceLimits.java | 8 +-
.../main/java/opennlp/tools/util/StringUtil.java | 97 ++--
.../java/opennlp/tools/util/WhitespaceMode.java | 23 +-
.../opennlp/tools/util/ext/ExtensionLoader.java | 11 +-
.../java/opennlp/tools/util/ext/ProviderSpec.java | 26 +-
.../java/opennlp/tools/util/ext/Providers.java | 17 +-
.../opennlp/tools/util/normalizer/AlignedText.java | 28 +-
.../opennlp/tools/util/normalizer/Alignment.java | 20 +-
.../opennlp/tools/util/normalizer/CharClass.java | 407 ++++++++--------
.../tools/util/normalizer/CodePointSet.java | 58 ++-
.../tools/util/normalizer/HexCodePoints.java | 26 +-
.../util/normalizer/OffsetAwareNormalizer.java | 2 +
.../opennlp/tools/util/normalizer/UnicodeDash.java | 38 +-
.../tools/util/normalizer/UnicodeWhitespace.java | 50 +-
.../tools/depparse/DependencySampleTest.java | 262 ++++++++++
.../opennlp/tools/depparse/DependencyTreeTest.java | 185 +++++++
.../tools/document/DocumentAnnotatorsTest.java | 12 +
.../tools/document/DocumentContractTest.java | 4 +
.../opennlp/tools/tokenize/MappedTextTest.java | 88 ++++
.../tools/tokenize/WordpieceEncoderTest.java | 37 +-
.../opennlp/tools/util/ArgumentChecksTest.java | 60 ---
.../java/opennlp/tools/util/ParamChecksTest.java | 240 +++++++++
.../tools/util/StringUtilSplitNonEmptyTest.java | 7 +-
.../java/opennlp/tools/util/StringUtilTest.java | 253 +++++-----
.../tools/util/normalizer/AlignedTextTest.java | 18 +
.../tools/util/normalizer/AlignmentTest.java | 8 +
.../tools/util/normalizer/CharClassTest.java | 111 +++--
.../tools/util/normalizer/CodePointSetTest.java | 51 ++
.../tools/cmdline/AbstractEvaluatorTool.java | 12 +
.../opennlp/tools/cmdline/AbstractTrainerTool.java | 7 +
.../src/main/java/opennlp/tools/cmdline/CLI.java | 10 +
.../tools/cmdline/DetailedFMeasureListener.java | 19 +
.../tools/cmdline/FineGrainedReportListener.java | 53 +-
.../DependencyEvaluationErrorListener.java} | 36 +-
.../DependencyModelLoader.java} | 18 +-
.../DependencyParserCrossValidatorTool.java | 79 +++
.../depparse/DependencyParserEvaluatorTool.java | 71 +++
.../cmdline/depparse/DependencyParserMETool.java | 89 ++++
.../depparse/DependencyParserTrainerTool.java | 80 +++
.../opennlp/tools/AbstractModelLoaderTest.java | 4 +-
.../ChunkerDetailedFMeasureListenerTest.java | 21 +-
.../depparse/DependencyParserToolsTest.java | 447 +++++++++++++++++
.../tools/cmdline/StreamFactoryRegistry.java | 2 +
.../java/opennlp/tools/formats/ad/ADMetadata.java | 29 +-
.../tools/formats/ad/ADNameSampleStream.java | 31 +-
.../tools/formats/ad/ADSentenceSampleStream.java | 8 +-
.../opennlp/tools/formats/ad/ADSentenceStream.java | 7 +-
.../tools/formats/brat/BratAnnotationStream.java | 7 +-
.../conllu/ConlluDependencySampleStream.java | 293 +++++++++++
.../ConlluDependencySampleStreamFactory.java | 109 +++++
.../conllu/ConlluLemmaSampleStreamFactory.java | 6 +-
.../conllu/ConlluPOSSampleStreamFactory.java | 6 +-
.../opennlp/tools/formats/conllu/ConlluStream.java | 10 +-
.../opennlp/tools/formats/conllu/ConlluTagset.java | 55 ++-
.../tools/formats/conllu/ConlluWordLine.java | 2 +-
.../tools/formats/ad/ADNameSampleStreamTest.java | 3 +-
.../formats/ad/ADSentenceSampleStreamTest.java | 5 +-
.../tools/formats/ad/ADSentenceStreamTest.java | 23 +
.../conllu/ConlluDependencyParserUsageTest.java | 214 ++++++++
.../ConlluDependencySampleStreamFactoryTest.java | 113 +++++
.../conllu/ConlluDependencySampleStreamTest.java | 453 +++++++++++++++++
.../tools/formats/conllu/ConlluTagsetTest.java | 63 +++
.../tools/formats/conllu/ConlluTestLines.java} | 21 +-
.../tools/formats/masc/MascParserTestUtil.java | 3 +-
opennlp-core/opennlp-ml/opennlp-dl-gpu/pom.xml | 26 +
opennlp-core/opennlp-ml/opennlp-dl/README.md | 27 +-
.../src/main/java/opennlp/dl/AbstractDL.java | 536 +++++++++++++++++----
.../src/main/java/opennlp/dl/JsonScan.java | 158 ++++--
.../dl/doccat/DocumentCategorizerConfig.java | 5 +-
.../opennlp/dl/doccat/DocumentCategorizerDL.java | 150 +++---
.../java/opennlp/dl/namefinder/NameFinderDL.java | 166 +++----
.../dl/vectors/OnnxTextEmbedderProvider.java | 17 +-
.../java/opennlp/dl/vectors/PaddingStrategy.java | 4 +-
.../java/opennlp/dl/vectors/SentenceVectorsDL.java | 168 +++----
.../java/opennlp/dl/AbstractDLChunkingTest.java | 24 +-
.../opennlp/dl/AbstractDLColdStartCudaTest.java | 67 ---
.../test/java/opennlp/dl/ColdStartCudaProbe.java | 50 --
.../src/test/java/opennlp/dl/GpuRequestTest.java | 100 ++++
.../opennlp/dl/InferenceOptionsValidationTest.java | 22 +
.../src/test/java/opennlp/dl/JsonScanTest.java | 57 +++
.../src/test/java/opennlp/dl/LoadVocabTest.java | 520 ++++++++++++++++++--
.../dl/doccat/DocumentCategorizerDLTest.java | 21 +-
.../opennlp/dl/namefinder/NameFinderDLTest.java | 18 +-
.../dl/vectors/OnnxTextEmbedderProviderTest.java | 11 +
.../vectors/SentenceVectorsDLBatchLimitTest.java | 296 ++++++++++++
.../dl/vectors/SentenceVectorsDLEmbedderTest.java | 213 ++++----
.../SentenceVectorsDLInferenceOptionsTest.java | 200 +++++---
.../vectors/SentenceVectorsDLPaddingEdgesTest.java | 337 +++++++++++++
.../java/opennlp/dl/vectors/TinyVectorModels.java | 197 ++++++++
.../src/test/resources/opennlp/dl/tokenizer.json | 79 +++
.../opennlp/dl/vectors/gen_tiny_masked_model.py | 8 +-
.../tools/ml/naivebayes/NaiveBayesModel.java | 2 +-
.../opennlp/tools/ml/AbstractMLModelWriter.java | 4 +-
.../src/main/java/opennlp/tools/ml/BeamSearch.java | 62 ++-
.../tools/ml/model/AbstractDataIndexer.java | 4 +-
.../java/opennlp/tools/ml/model/AbstractModel.java | 2 +-
.../java/opennlp/tools/ml/model/EventFields.java | 37 +-
.../opennlp/tools/ml/model/FileEventStream.java | 8 +-
.../tools/ml/model/RealValueFileEventStream.java | 45 +-
.../tools/ml/model/SimpleEventStreamBuilder.java | 29 +-
.../java/opennlp/tools/ml/model/EventTest.java | 20 +-
.../ml/model/RealValueFileEventStreamTest.java | 38 +-
.../ml/model/SimpleEventStreamBuilderTest.java | 43 +-
.../tools/ml/libsvm/doccat/SvmDoccatModel.java | 14 +-
.../tools/ml/maxent/BasicContextGenerator.java | 18 +-
.../tools/ml/maxent/RealBasicEventStream.java | 11 +-
.../tools/ml/maxent/BasicContextGeneratorTest.java | 3 +-
.../tools/ml/maxent/RealBasicEventStreamTest.java | 37 +-
.../tools/ml/perceptron/PerceptronModel.java | 2 +-
.../tools/models/AbstractClassPathModelFinder.java | 9 +-
opennlp-core/opennlp-runtime/BENCHMARKS.md | 12 +
.../java/opennlp/tools/ml/BeamSearchBenchmark.java | 11 +-
.../opennlp/tools/chunker/ChunkerAnnotator.java | 5 +-
.../java/opennlp/tools/chunker/ChunkerModel.java | 4 +-
.../opennlp/tools/depparse/ArcStandardOracle.java | 148 ++++++
.../opennlp/tools/depparse/ArcStandardState.java | 303 ++++++++++++
.../opennlp/tools/depparse/AttachmentScores.java | 114 +++++
.../tools/depparse/DependencyContextGenerator.java | 348 +++++++++++++
.../tools/depparse/DependencyCrossValidator.java | 170 +++++++
.../tools/depparse/DependencyEvaluator.java | 157 ++++++
.../tools/depparse/DependencyEventStream.java | 126 +++++
.../opennlp/tools/depparse/DependencyModel.java | 125 +++++
.../opennlp/tools/depparse/DependencyParserME.java | 208 ++++++++
.../java/opennlp/tools/depparse/Transition.java | 129 +++++
.../tools/dictionary/serializer/Attributes.java | 8 +-
.../tools/doccat/BagOfWordsFeatureGenerator.java | 9 +-
.../java/opennlp/tools/doccat/DoccatModel.java | 4 +-
.../tools/doccat/EmojiFeatureGenerator.java | 5 +-
.../tools/doccat/NGramFeatureGenerator.java | 9 +-
.../tools/entitylinker/EntityLinkerFactory.java | 4 +-
.../tools/lemmatizer/LemmatizerAnnotator.java | 8 +-
.../java/opennlp/tools/log/LogPrintStream.java | 8 +-
.../namefind/DefaultNameContextGenerator.java | 6 +-
.../tools/namefind/DictionaryNameFinder.java | 7 +-
.../tools/namefind/NameFinderAnnotator.java | 8 +-
.../opennlp/tools/namefind/RegexNameFinder.java | 24 +-
.../tools/namefind/RegexNameFinderFactory.java | 22 +-
.../java/opennlp/tools/parser/ParserAnnotator.java | 9 +-
.../java/opennlp/tools/parser/ParserModel.java | 10 +-
.../parser/lang/es/AncoraSpanishHeadRules.java | 13 +-
.../postag/ConfigurablePOSContextGenerator.java | 7 +-
.../java/opennlp/tools/postag/POSDictionary.java | 6 +-
.../main/java/opennlp/tools/postag/POSModel.java | 10 +-
.../opennlp/tools/postag/POSTagFormatMapper.java | 15 +-
.../opennlp/tools/postag/POSTaggerAnnotator.java | 8 +-
.../sentdetect/SentenceDetectorAnnotator.java | 12 +-
.../tools/sentdetect/SentenceDetectorME.java | 23 +-
.../opennlp/tools/sentdetect/SentenceModel.java | 4 +-
.../sentiment/EmojiSentimentContextGenerator.java | 5 +-
.../tools/sentiment/EmojiSentimentFactory.java | 2 +
.../java/opennlp/tools/stemmer/CachingStemmer.java | 18 +-
.../opennlp/tools/stemmer/DelegatingStemmer.java | 21 +-
.../java/opennlp/tools/stemmer/PorterStemmer.java | 14 +-
.../java/opennlp/tools/stemmer/SharingStemmer.java | 13 +-
.../opennlp/tools/stemmer/StemmerAnnotator.java | 8 +-
.../tools/stemmer/StemmerAnnotatorProvider.java | 9 +-
.../tools/stemmer/hunspell/AffixCondition.java | 20 +-
.../tools/stemmer/hunspell/HunspellDictionary.java | 80 +--
...nload.java => HunspellDictionaryInstaller.java} | 27 +-
.../tools/stemmer/hunspell/HunspellStemmer.java | 13 +-
.../stemmer/hunspell/HunspellStemmerFactory.java | 5 +-
.../tools/stemmer/snowball/SnowballStemmer.java | 3 +-
.../stemmer/snowball/SnowballStemmerFactory.java | 15 +-
.../tools/stopword/DictionaryStopwordFilter.java | 33 +-
.../tools/stopword/StopwordFilterStream.java | 6 +-
.../tools/stopword/StopwordFilteringTokenizer.java | 9 +-
.../java/opennlp/tools/stopword/StopwordLists.java | 13 +-
.../tools/termvector/TermVectorAnnotator.java | 50 +-
.../java/opennlp/tools/tokenize/BPETokenizer.java | 13 +-
.../tools/tokenize/BPETokenizerTrainer.java | 11 +-
.../opennlp/tools/tokenize/TokSpanEventStream.java | 4 +
.../opennlp/tools/tokenize/TokenSampleStream.java | 8 +-
.../opennlp/tools/tokenize/TokenizerAnnotator.java | 14 +-
.../tools/tokenize/TokenizerCharacterPolicy.java | 174 +++++++
.../opennlp/tools/tokenize/TokenizerFactory.java | 7 +
.../java/opennlp/tools/tokenize/TokenizerME.java | 3 +
.../opennlp/tools/tokenize/TokenizerModel.java | 4 +-
.../java/opennlp/tools/tokenize/lang/Factory.java | 34 +-
.../tools/tokenize/lattice/CategoryTable.java | 15 +-
.../tools/tokenize/lattice/DoubleArrayLexicon.java | 10 +-
.../tools/tokenize/lattice/LatticeTokenizer.java | 26 +-
.../tools/tokenize/lattice/MecabDictionary.java | 167 ++++---
.../tokenize/lattice/MecabDictionaryInstaller.java | 175 ++-----
.../opennlp/tools/tokenize/lattice/Morpheme.java | 29 +-
.../tools/tokenize/lattice/UnigramSegmenter.java | 79 +--
.../tools/tokenize/lattice/WhitespaceRuns.java | 65 +++
.../tools/tokenize/uax29/ExtendedPictographic.java | 91 ++--
.../tools/tokenize/uax29/WordBreakProperty.java | 110 ++---
.../tools/tokenize/uax29/WordSegmenter.java | 18 +-
.../tools/tokenize/uax29/WordTokenizer.java | 28 +-
.../opennlp/tools/tokenize/uax29/WordType.java | 14 +-
.../main/java/opennlp/tools/util/Checksums.java | 96 ++++
.../java/opennlp/tools/util/DictionaryCatalog.java | 92 ++--
.../main/java/opennlp/tools/util/DownloadUtil.java | 158 ++++--
.../opennlp/tools/util/FilterObjectStream.java | 4 +-
.../java/opennlp/tools/util/ResourceInstaller.java | 308 ++++++------
.../main/java/opennlp/tools/util/StringList.java | 3 +-
.../java/opennlp/tools/util/archive/TarStream.java | 74 ++-
.../featuregen/AggregatedFeatureGenerator.java | 15 +-
.../EmojiAnnotationFeatureGenerator.java | 5 +-
.../EmojiAnnotationFeatureGeneratorFactory.java | 1 +
.../tools/util/featuregen/InSpanGenerator.java | 7 +-
.../java/opennlp/tools/util/model/BaseModel.java | 17 +-
.../tools/util/model/GenericModelSerializer.java | 11 +-
.../java/opennlp/tools/util/model/ModelUtil.java | 6 +-
.../AccentFoldCharSequenceNormalizer.java | 15 +-
.../AggregateCharSequenceNormalizer.java | 6 +-
.../AlignedAggregateCharSequenceNormalizer.java | 4 +
.../opennlp/tools/util/normalizer/AsciiChars.java | 19 +
.../normalizer/BulletCharSequenceNormalizer.java | 14 +-
.../tools/util/normalizer/BundledUnicodeData.java | 204 ++++++++
.../normalizer/CaseFoldCharSequenceNormalizer.java | 16 +-
.../ConfusableSkeletonCharSequenceNormalizer.java | 9 +-
.../opennlp/tools/util/normalizer/Confusables.java | 104 ++--
.../normalizer/DashCharSequenceNormalizer.java | 7 +
.../normalizer/DigitCharSequenceNormalizer.java | 8 +-
.../opennlp/tools/util/normalizer/Dimension.java | 10 +-
.../normalizer/EllipsisCharSequenceNormalizer.java | 5 +-
.../tools/util/normalizer/EmojiAnnotation.java | 19 +-
.../tools/util/normalizer/EmojiAnnotations.java | 113 +++--
.../tools/util/normalizer/EmojiAnnotator.java | 19 +-
.../normalizer/EmojiCharSequenceNormalizer.java | 4 +-
.../tools/util/normalizer/EmojiEmoticons.java | 134 ++----
.../opennlp/tools/util/normalizer/EmojiFlags.java | 13 +-
.../EmojiToEmoticonCharSequenceNormalizer.java | 11 +-
.../EmoticonToEmojiCharSequenceNormalizer.java | 11 +-
.../FullCaseFoldCharSequenceNormalizer.java | 117 +++--
.../GermanUmlautCharSequenceNormalizer.java | 10 +-
.../InvisibleCharSequenceNormalizer.java | 11 +-
...PreservingWhitespaceCharSequenceNormalizer.java | 2 +-
.../tools/util/normalizer/MailAddressScan.java | 4 +-
.../util/normalizer/NfcCharSequenceNormalizer.java | 11 +-
.../normalizer/NfkcCharSequenceNormalizer.java | 11 +-
.../util/normalizer/NormalizationProfile.java | 25 +-
.../util/normalizer/NormalizationProfiles.java | 32 +-
.../normalizer/NumberCharSequenceNormalizer.java | 4 +-
...PreservingWhitespaceCharSequenceNormalizer.java | 3 +-
.../normalizer/QuoteCharSequenceNormalizer.java | 8 +-
.../normalizer/ShrinkCharSequenceNormalizer.java | 4 +-
.../SocialMediaCharSequenceNormalizer.java | 9 +-
.../java/opennlp/tools/util/normalizer/Term.java | 13 +-
.../tools/util/normalizer/TermAnalyzer.java | 34 +-
.../tools/util/normalizer/TextNormalizer.java | 13 +-
.../util/normalizer/UnicodeEmojiSequences.java | 83 ++--
.../util/normalizer/UrlCharSequenceNormalizer.java | 34 +-
.../WhitespaceCharSequenceNormalizer.java | 2 +-
.../opennlp/tools/AbstractModelLoaderTest.java | 4 +-
.../opennlp/tools/NullArgumentRejectionTest.java | 210 ++++++++
.../tools/chunker/ChunkerAnnotatorTest.java | 18 +-
.../tools/depparse/ArcStandardOracleTest.java | 195 ++++++++
.../tools/depparse/ArcStandardStateTest.java | 135 ++++++
.../tools/depparse/AttachmentScoresTest.java | 77 +++
.../depparse/DependencyContextGeneratorTest.java | 235 +++++++++
.../depparse/DependencyCrossValidatorTest.java | 268 +++++++++++
.../tools/depparse/DependencyEvaluatorTest.java | 250 ++++++++++
.../depparse/DependencyParserEdgeCaseTest.java | 178 +++++++
.../tools/depparse/DependencyParserMETest.java | 314 ++++++++++++
.../tools/depparse/DependencyTestSamples.java | 141 ++++++
.../opennlp/tools/depparse/RecordingMonitor.java | 48 ++
.../opennlp/tools/depparse/TransitionTest.java | 70 +++
.../doccat/BagOfWordsFeatureGeneratorTest.java | 4 +-
.../opennlp/tools/doccat/DocumentSampleTest.java | 9 +
.../tools/doccat/NGramFeatureGeneratorTest.java | 4 +-
.../tools/document/DocumentAnalyzerTest.java | 49 +-
.../document/DocumentPipelineExampleTest.java | 10 +-
...{TestComponents.java => DocumentTestStubs.java} | 25 +-
.../tools/langdetect/LanguageSampleTest.java | 4 +-
.../opennlp/tools/langdetect/LanguageTest.java | 4 +-
.../tools/namefind/NameFinderMEWithDatesTest.java | 2 +-
.../opennlp/tools/namefind/NameSampleTest.java | 74 +++
.../test/java/opennlp/tools/parser/ParseTest.java | 181 +++++++
.../opennlp/tools/parser/ParserAnnotatorTest.java | 15 +-
.../lang/es/AncoraSpanishHeadRulesBundledTest.java | 147 ++++++
.../opennlp/tools/postag/POSDictionaryTest.java | 10 +
.../opennlp/tools/stemmer/StemmerFactoryTest.java | 4 +
...t.java => HunspellDictionaryInstallerTest.java} | 12 +-
.../tools/termvector/SingleSpaceTokens.java | 46 +-
.../tools/termvector/TermVectorAnnotatorTest.java | 5 +
.../tools/termvector/TermVectorPipelineTest.java | 5 +-
.../opennlp/tools/tokenize/TokenSampleTest.java | 23 +
.../tokenize/TokenizerCharacterPolicyTest.java | 256 ++++++++++
.../opennlp/tools/tokenize/TokenizerMETest.java | 6 -
.../tools/tokenize/WhitespaceTokenizerTest.java | 7 +-
.../tools/tokenize/WordpieceTokenizerTest.java | 29 +-
.../tokenize/lattice/LatticeTokenizerTest.java | 58 ++-
.../lattice/MecabDictionaryInstallerTest.java | 7 +-
.../tokenize/lattice/UnigramSegmenterTest.java | 6 +-
.../tokenize/uax29/WordBreakPropertyTest.java | 2 +-
.../tools/tokenize/uax29/WordSegmenterTest.java | 5 +-
.../tools/tokenize/uax29/WordTokenizerTest.java | 5 +-
.../opennlp/tools/tokenize/uax29/WordTypeTest.java | 5 +-
.../java/opennlp/tools/util/ChecksumsTest.java | 85 ++++
.../opennlp/tools/util/DictionaryCatalogTest.java | 87 ++++
.../opennlp/tools/util/InstallerTestSupport.java | 42 --
.../tools/util/ResourceInstallerHttpTest.java | 16 +-
.../tools/util/ResourceInstallerSelectedTest.java | 198 ++++++++
.../opennlp/tools/util/ResourceInstallerTest.java | 61 ++-
.../opennlp/tools/util/archive/TarStreamTest.java | 58 +++
.../AccentFoldCharSequenceNormalizerTest.java | 5 +-
.../normalizer/AlignedNormalizerPipelineTest.java | 9 +-
.../util/normalizer/BundledUnicodeDataTest.java | 161 +++++++
.../CaseFoldCharSequenceNormalizerTest.java | 9 +-
.../CharSequenceNormalizerContractTest.java | 33 ++
.../util/normalizer/ConfusableSkeletonTest.java | 5 +-
.../tools/util/normalizer/ConfusablesTest.java | 5 +-
...CharSequenceNormalizerCharacterizationTest.java | 5 +-
.../EmojiCharSequenceNormalizerTest.java | 6 +-
.../EmojiToEmoticonCharSequenceNormalizerTest.java | 5 +-
.../EmoticonToEmojiCharSequenceNormalizerTest.java | 5 +-
.../FullCaseFoldCharSequenceNormalizerTest.java | 5 +-
.../GermanUmlautCharSequenceNormalizerTest.java | 5 +-
...ervingWhitespaceCharSequenceNormalizerTest.java | 5 +-
.../util/normalizer/NormalizationProfilesTest.java | 13 +-
...ervingWhitespaceCharSequenceNormalizerTest.java | 5 +-
.../util/normalizer/SetBasedNormalizerTest.java | 5 +-
.../normalizer/TermAnalyzerMultilingualTest.java | 5 +-
.../tools/util/normalizer/TermAnalyzerTest.java | 9 +-
.../tools/util/normalizer/TextNormalizerTest.java | 5 +-
.../UnicodeCharSequenceNormalizerTest.java | 5 +-
.../normalizer/UnicodeEmojiSequencesParseTest.java | 5 +-
.../resources/opennlp/tools/parser/es_head_rules | 22 +
opennlp-distr/pom.xml | 26 +-
opennlp-docs/src/docbkx/corpora.xml | 22 +-
opennlp-docs/src/docbkx/dependency.xml | 135 ++++++
opennlp-docs/src/docbkx/doccat.xml | 12 +-
opennlp-docs/src/docbkx/document.xml | 42 +-
opennlp-docs/src/docbkx/introduction.xml | 47 +-
opennlp-docs/src/docbkx/langdetect.xml | 13 +-
opennlp-docs/src/docbkx/machine-learning.xml | 24 +-
opennlp-docs/src/docbkx/morfologik-addon.xml | 6 +
opennlp-docs/src/docbkx/namefinder.xml | 5 +-
opennlp-docs/src/docbkx/normalizer.xml | 80 +--
opennlp-docs/src/docbkx/opennlp.xml | 1 +
opennlp-docs/src/docbkx/spellcheck.xml | 14 +
opennlp-docs/src/docbkx/stemmer.xml | 32 +-
opennlp-docs/src/docbkx/tokenizer.xml | 88 ++--
opennlp-eval-tests/lang/es/parser/es-head-rules | 24 +-
.../dl/doccat/DocumentCategorizerDLEval.java | 3 +-
.../opennlp/dl/namefinder/NameFinderDLEval.java | 98 ++--
.../java/opennlp/tools/eval/AbstractEvalTest.java | 29 ++
.../tools/eval/HunspellClassificationEval.java | 80 +++
.../tools/eval/HunspellCompatibilityEval.java | 43 +-
.../tools/eval/MultiThreadedStemmerEval.java | 94 +++-
.../tools/eval/UniversalDependencyParserEval.java | 301 ++++++++++++
.../builder/MorfologikDictionaryBuilder.java | 14 +-
.../lemmatizer/MorfologikLemmatizer.java | 2 +
.../tagdict/MorfologikTagDictionary.java | 37 +-
.../opennlp/morfologik/AbstractMorfologikTest.java | 6 +-
.../MorfologikTagDictionaryConcurrencyTest.java | 94 ++++
.../tagdict/MorfologikTagDictionaryTest.java | 39 ++
...aryLocaleSafety.info => dictionaryTagless.info} | 0
.../src/test/resources/dictionaryTagless.txt | 2 +
.../main/java/opennlp/spellcheck/SpellChecker.java | 11 +-
.../main/java/opennlp/spellcheck/SuggestItem.java | 9 +-
.../dictionary/FrequencyDictionaryLoader.java | 15 +-
.../spellcheck/dictionary/SymSpellModel.java | 26 +-
.../dictionary/SymSpellModelResolver.java | 16 +-
.../spellcheck/dictionary/SymSpellModels.java | 18 +-
.../SpellCheckingCharSequenceNormalizer.java | 89 ++--
.../stream/SpellCorrectingObjectStream.java | 10 +-
.../stream/SpellCorrectingTokenStream.java | 12 +-
.../java/opennlp/spellcheck/symspell/SymSpell.java | 25 +-
.../spellcheck/symspell/SymSpellConfig.java | 6 +-
.../java/opennlp/spellcheck/SuggestItemTest.java | 4 +-
.../dictionary/FrequencyDictionaryLoaderTest.java | 18 +
.../dictionary/SymSpellModelResolverTest.java | 11 +
.../dictionary/SymSpellModelSerializationTest.java | 28 ++
.../SpellCheckingCharSequenceNormalizerTest.java | 60 +--
.../normalizer/SpellCheckingUnicodeCaseTest.java | 116 +++++
.../spellcheck/symspell/SymSpellCompoundTest.java | 2 +-
.../spellcheck/symspell/SymSpellConfigTest.java | 2 +-
.../spellcheck/symspell/SymSpellLookupTest.java | 14 +-
.../java/opennlp/uima/normalizer/NumberUtil.java | 7 +-
.../src/test/java/opennlp/uima/AbstractIT.java | 4 +-
.../opennlp/uima/normalizer/NumberUtilTest.java | 26 +-
opennlp-tools/lang/es/parser/es-head-rules | 24 +-
.../tools/namefind/AbstractNameFinderTest.java | 4 +-
.../opennlp/tools/util/DownloadParserTest.java | 109 ++++-
.../opennlp/tools/util/ObjectStreamUtilsTest.java | 21 +
.../tools/util/PlainTextByLineStreamTest.java | 9 +
.../tools/util/ext/ExtensionLoaderTest.java | 8 +-
pom.xml | 21 +-
rat-excludes | 1 +
429 files changed, 17650 insertions(+), 4489 deletions(-)
create mode 100644 dev/forbidden-regex.txt
create mode 100644
opennlp-api/src/main/java/opennlp/tools/commons/SuppressForbidden.java
create mode 100644
opennlp-api/src/main/java/opennlp/tools/depparse/DependencyArc.java
copy
opennlp-api/src/main/java/opennlp/tools/{lemmatizer/LemmatizerEvaluationMonitor.java
=> depparse/DependencyEvaluationMonitor.java} (79%)
create mode 100644
opennlp-api/src/main/java/opennlp/tools/depparse/DependencyParser.java
create mode 100644
opennlp-api/src/main/java/opennlp/tools/depparse/DependencySample.java
create mode 100644
opennlp-api/src/main/java/opennlp/tools/depparse/DependencyTree.java
create mode 100644
opennlp-api/src/main/java/opennlp/tools/document/DocumentMerges.java
create mode 100644
opennlp-api/src/main/java/opennlp/tools/tokenize/MappedText.java
delete mode 100644
opennlp-api/src/main/java/opennlp/tools/util/ArgumentChecks.java
create mode 100644
opennlp-api/src/main/java/opennlp/tools/util/ParamChecks.java
create mode 100644
opennlp-api/src/test/java/opennlp/tools/depparse/DependencySampleTest.java
create mode 100644
opennlp-api/src/test/java/opennlp/tools/depparse/DependencyTreeTest.java
create mode 100644
opennlp-api/src/test/java/opennlp/tools/tokenize/MappedTextTest.java
delete mode 100644
opennlp-api/src/test/java/opennlp/tools/util/ArgumentChecksTest.java
create mode 100644
opennlp-api/src/test/java/opennlp/tools/util/ParamChecksTest.java
copy
opennlp-core/opennlp-cli/src/main/java/opennlp/tools/cmdline/{tokenizer/DetokenEvaluationErrorListener.java
=> depparse/DependencyEvaluationErrorListener.java} (51%)
copy
opennlp-core/opennlp-cli/src/main/java/opennlp/tools/cmdline/{lemmatizer/LemmatizerModelLoader.java
=> depparse/DependencyModelLoader.java} (68%)
create mode 100644
opennlp-core/opennlp-cli/src/main/java/opennlp/tools/cmdline/depparse/DependencyParserCrossValidatorTool.java
create mode 100644
opennlp-core/opennlp-cli/src/main/java/opennlp/tools/cmdline/depparse/DependencyParserEvaluatorTool.java
create mode 100644
opennlp-core/opennlp-cli/src/main/java/opennlp/tools/cmdline/depparse/DependencyParserMETool.java
create mode 100644
opennlp-core/opennlp-cli/src/main/java/opennlp/tools/cmdline/depparse/DependencyParserTrainerTool.java
create mode 100644
opennlp-core/opennlp-cli/src/test/java/opennlp/tools/cmdline/depparse/DependencyParserToolsTest.java
create mode 100644
opennlp-core/opennlp-formats/src/main/java/opennlp/tools/formats/conllu/ConlluDependencySampleStream.java
create mode 100644
opennlp-core/opennlp-formats/src/main/java/opennlp/tools/formats/conllu/ConlluDependencySampleStreamFactory.java
create mode 100644
opennlp-core/opennlp-formats/src/test/java/opennlp/tools/formats/conllu/ConlluDependencyParserUsageTest.java
create mode 100644
opennlp-core/opennlp-formats/src/test/java/opennlp/tools/formats/conllu/ConlluDependencySampleStreamFactoryTest.java
create mode 100644
opennlp-core/opennlp-formats/src/test/java/opennlp/tools/formats/conllu/ConlluDependencySampleStreamTest.java
create mode 100644
opennlp-core/opennlp-formats/src/test/java/opennlp/tools/formats/conllu/ConlluTagsetTest.java
copy
opennlp-core/opennlp-formats/src/{main/java/opennlp/tools/formats/conllu/ConlluTagset.java
=> test/java/opennlp/tools/formats/conllu/ConlluTestLines.java} (64%)
delete mode 100644
opennlp-core/opennlp-ml/opennlp-dl/src/test/java/opennlp/dl/AbstractDLColdStartCudaTest.java
delete mode 100644
opennlp-core/opennlp-ml/opennlp-dl/src/test/java/opennlp/dl/ColdStartCudaProbe.java
create mode 100644
opennlp-core/opennlp-ml/opennlp-dl/src/test/java/opennlp/dl/GpuRequestTest.java
create mode 100644
opennlp-core/opennlp-ml/opennlp-dl/src/test/java/opennlp/dl/vectors/SentenceVectorsDLBatchLimitTest.java
create mode 100644
opennlp-core/opennlp-ml/opennlp-dl/src/test/java/opennlp/dl/vectors/SentenceVectorsDLPaddingEdgesTest.java
create mode 100644
opennlp-core/opennlp-ml/opennlp-dl/src/test/java/opennlp/dl/vectors/TinyVectorModels.java
create mode 100644
opennlp-core/opennlp-ml/opennlp-dl/src/test/resources/opennlp/dl/tokenizer.json
create mode 100644
opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/depparse/ArcStandardOracle.java
create mode 100644
opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/depparse/ArcStandardState.java
create mode 100644
opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/depparse/AttachmentScores.java
create mode 100644
opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/depparse/DependencyContextGenerator.java
create mode 100644
opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/depparse/DependencyCrossValidator.java
create mode 100644
opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/depparse/DependencyEvaluator.java
create mode 100644
opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/depparse/DependencyEventStream.java
create mode 100644
opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/depparse/DependencyModel.java
create mode 100644
opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/depparse/DependencyParserME.java
create mode 100644
opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/depparse/Transition.java
rename
opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/hunspell/{HunspellDictionaryDownload.java
=> HunspellDictionaryInstaller.java} (75%)
create mode 100644
opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/tokenize/TokenizerCharacterPolicy.java
create mode 100644
opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/tokenize/lattice/WhitespaceRuns.java
create mode 100644
opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/Checksums.java
create mode 100644
opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/BundledUnicodeData.java
create mode 100644
opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/NullArgumentRejectionTest.java
create mode 100644
opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/depparse/ArcStandardOracleTest.java
create mode 100644
opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/depparse/ArcStandardStateTest.java
create mode 100644
opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/depparse/AttachmentScoresTest.java
create mode 100644
opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/depparse/DependencyContextGeneratorTest.java
create mode 100644
opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/depparse/DependencyCrossValidatorTest.java
create mode 100644
opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/depparse/DependencyEvaluatorTest.java
create mode 100644
opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/depparse/DependencyParserEdgeCaseTest.java
create mode 100644
opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/depparse/DependencyParserMETest.java
create mode 100644
opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/depparse/DependencyTestSamples.java
create mode 100644
opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/depparse/RecordingMonitor.java
create mode 100644
opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/depparse/TransitionTest.java
rename
opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/document/{TestComponents.java
=> DocumentTestStubs.java} (76%)
create mode 100644
opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/parser/lang/es/AncoraSpanishHeadRulesBundledTest.java
rename
opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/hunspell/{HunspellDictionaryDownloadTest.java
=> HunspellDictionaryInstallerTest.java} (94%)
create mode 100644
opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/tokenize/TokenizerCharacterPolicyTest.java
create mode 100644
opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/ChecksumsTest.java
create mode 100644
opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/ResourceInstallerSelectedTest.java
create mode 100644
opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/normalizer/BundledUnicodeDataTest.java
create mode 100644
opennlp-core/opennlp-runtime/src/test/resources/opennlp/tools/parser/es_head_rules
create mode 100644 opennlp-docs/src/docbkx/dependency.xml
create mode 100644
opennlp-eval-tests/src/test/java/opennlp/tools/eval/HunspellClassificationEval.java
create mode 100644
opennlp-eval-tests/src/test/java/opennlp/tools/eval/UniversalDependencyParserEval.java
create mode 100644
opennlp-extensions/opennlp-morfologik/src/test/java/opennlp/morfologik/tagdict/MorfologikTagDictionaryConcurrencyTest.java
copy
opennlp-extensions/opennlp-morfologik/src/test/resources/{dictionaryLocaleSafety.info
=> dictionaryTagless.info} (100%)
create mode 100644
opennlp-extensions/opennlp-morfologik/src/test/resources/dictionaryTagless.txt
create mode 100644
opennlp-extensions/opennlp-spellcheck/src/test/java/opennlp/spellcheck/normalizer/SpellCheckingUnicodeCaseTest.java
