This is an automated email from the ASF dual-hosted git repository.
rzo1 pushed a change to branch OPENNLP-1982-sentence-vectors-inference-options
in repository https://gitbox.apache.org/repos/asf/opennlp.git
omit 3d7395de4 SentenceVectorsDL cannot run on a GPU
omit 756e58ee8 GPU inference fails on the first CUDA request in a fresh JVM
add 47369544c (fix) GH-1378: MeCab CSV reader accepts unterminated quoted
fields (#1383)
add a368de9cd (fix) GH-1379: Morpheme accepts a surface that does not
match its span (#1384)
add 197933cf9 OPENNLP-1934: Add explicit tokenizer character policies
(#1349)
add 42877392f Minor: Use ArgumentChecks for inline null checks in
normalizer and tokenize (#1400)
add fce5fdbd6 Minor: Use ArgumentChecks for the inline null checks in
opennlp-api (#1399)
add c8536d2c1 Minor: Use ArgumentChecks for inline null checks in ml,
model-resolver, spellcheck (#1402)
add 35e97920d Minor: Use ArgumentChecks for the remaining inline null
checks in runtime (#1401)
add d742b18f0 MorfologikTagDictionary is not thread-safe and fails on
entries without a tag (#1382)
add b2e843340 Bump org.apache.maven:apache-maven from 3.9.16 to 3.10.0
(#1405)
add dd45c0042 Bump com.puppycrawl.tools:checkstyle from 13.3.0 to 14.3.0
(#1407)
add a6871186b GH-1408: Rename ArgumentChecks to ParamChecks (#1409)
add 6c8fdd5c7 GH-1397: Reject null spellcheck arguments with
IllegalArgumentException (#1412)
add 169827669 GH-1396: Use ParamChecks for null arguments in
opennlp-runtime (#1411)
add 66b1719bd GH-1395: Reject null arguments in opennlp-api with
IllegalArgumentException (#1410)
add 19f201e80 GH-1414: Store mvnw.cmd with LF line endings (#1415)
add c98b76298 GH-1398: Add requireNonEmpty and requireNonNegative to
ParamChecks (#1413)
add 1fd51808e Spellcheck correction loses the capital of a
supplementary-plane initial (#1385)
add 8b8a4dc86 GH-1422: Clean up opennlp-morfologik builder and Javadoc
(#1430)
add af742d8bd GH-1422: Clean up NumberUtil and its tests in opennlp-uima
(#1432)
add e1a807c47 GH-1388: Load bundled Unicode data through one recoverable
loader (#1373)
add 42f446a5a GH-1422: Clean up the spellcheck normalizer and loader
(#1431)
add 95b1ac5ff GH-1422: Clean up the report listeners in opennlp-cli (#1429)
add d7eadeda2 GH-1422: Clean up the stemmer, Hunspell and NameFinderDL
evals (#1434)
add 0a363d573 GH-1421: Clean up the AD, MASC, CoNLL-U and Brat readers
(#1428)
add 97e4fb13f GH-1418: Clean up event parsing and context generators in
the ml modules (#1425)
add 0ca246230 GH-1417: Remove duplicated logic in opennlp-api and the
lattice tokenizer (#1424)
add ef04afe6d GH-1416: Correct and trim opennlp-api Javadoc (#1423)
add 9a6a927a5 GH-1422: Remove duplication in StringUtilTest and the
chunker listener test (#1433)
add 11a136cc6 GH-1441: Report malformed lattice dictionary content as
InvalidFormatException (#1443)
add bca31bb7c OPENNLP-1988: Load WordPiece vocabularies from
tokenizer.json (#1319)
add 013b789b1 GH-1394: Load downloaded models by language when the model
host cannot be reached (#1435)
add 5a6347ac2 GH-1436: Document thread-safety limits of
WhitespaceTokenizer.INSTANCE and the process-wide modes (#1437)
add cf81387cc GH-1438: Share code point validation and trim scans in
CharClass and CodePointSet (#1439)
add 617d38996 GH-1440: Isolate WordpieceTokenizer from the shared
WhitespaceTokenizer and share punctuation isolation (#1442)
add a9d226ce9 GH-1445: Run the CI build once per job and render
README.html from README.md only (#1446)
add e039a7fb7 GH-1393: Cache the downloaded test models between CI runs
(#1444)
add deb80bae8 OPENNLP-547: Add a transition-based dependency parser
component (#1236)
add 139b9ddb3 GH-1449: Share ONNX inference between DocumentCategorizerDL
and NameFinderDL with one failure message (#1450)
add ce2da5357 GH-1447: Reject self-closing AD markup tags as opening tags
(#1448)
add 779420863 GH-1451: Share staged install, checksum and tar helpers
across the dictionary installers (#1452)
add 8d2a8c310 GH-1419: Correct and trim opennlp-runtime Javadoc (#1426)
add eef3909dc GH-1420: Remove duplication and add readResolve in
opennlp-runtime (#1427)
add ef814bdba GH-1454: Tidy document, stemmer, corpora and doccat chapters
and README (#1455)
add 83d40b389 GPU inference fails on the first CUDA request in a fresh JVM
add a9d0437db SentenceVectorsDL cannot run on a GPU
add fb06aefd7 OPENNLP-1982: Keep the Javadoc of the CUDA changes to the
contract
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (3d7395de4)
\
N -- N -- N
refs/heads/OPENNLP-1982-sentence-vectors-inference-options (fb06aefd7)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.github/workflows/maven.yml | 19 +-
.mvn/wrapper/maven-wrapper.properties | 2 +-
README.md | 10 +-
dev/README-hunspell-dictionaries.md | 4 +-
.../java/opennlp/tools/depparse/DependencyArc.java | 65 +++
.../DependencyEvaluationMonitor.java} | 8 +-
.../opennlp/tools/depparse/DependencyParser.java | 50 ++
.../opennlp/tools/depparse/DependencySample.java | 145 ++++++
.../opennlp/tools/depparse/DependencyTree.java | 270 +++++++++++
.../java/opennlp/tools/doccat/DocumentSample.java | 7 +-
.../java/opennlp/tools/document/Annotation.java | 5 +-
.../main/java/opennlp/tools/document/Document.java | 42 +-
.../opennlp/tools/document/DocumentAnalyzer.java | 6 +-
.../opennlp/tools/document/DocumentAnnotator.java | 5 +-
.../opennlp/tools/document/DocumentAnnotators.java | 27 +-
.../opennlp/tools/document/DocumentMerges.java | 71 +++
.../opennlp/tools/document/ImmutableDocument.java | 41 +-
.../main/java/opennlp/tools/document/LayerKey.java | 5 +-
.../main/java/opennlp/tools/document/Layers.java | 24 +-
.../opennlp/tools/embeddings/TextEmbedder.java | 6 +-
.../java/opennlp/tools/langdetect/Language.java | 6 +-
.../opennlp/tools/langdetect/LanguageSample.java | 15 +-
.../main/java/opennlp/tools/ml/model/Event.java | 13 +-
.../java/opennlp/tools/namefind/NameSample.java | 6 +-
.../tools/namefind/OffsetMappingNameFinder.java | 13 +-
.../src/main/java/opennlp/tools/parser/Parse.java | 5 +-
.../opennlp/tools/sentiment/SentimentSample.java | 9 +-
.../java/opennlp/tools/stemmer/StemmerFactory.java | 4 +-
.../java/opennlp/tools/termvector/TermVector.java | 13 +-
.../opennlp/tools/tokenize/BertNormalization.java | 49 +-
.../java/opennlp/tools/tokenize/MappedText.java | 161 +++++++
.../java/opennlp/tools/tokenize/SubwordPiece.java | 14 +-
.../opennlp/tools/tokenize/SubwordTokenizer.java | 5 +
.../java/opennlp/tools/tokenize/TokenSample.java | 19 +-
.../tools/tokenize/WhitespaceTokenizer.java | 8 +-
.../opennlp/tools/tokenize/WordpieceEncoder.java | 195 ++------
.../opennlp/tools/tokenize/WordpieceTokenizer.java | 62 +--
.../java/opennlp/tools/util/ArgumentChecks.java | 57 ---
.../java/opennlp/tools/util/CompatibilityMode.java | 24 +-
.../opennlp/tools/util/LanguageCodeValidator.java | 13 +-
.../java/opennlp/tools/util/ObjectStreamUtils.java | 12 +-
.../main/java/opennlp/tools/util/ParamChecks.java | 223 +++++++++
.../opennlp/tools/util/PlainTextByLineStream.java | 7 +-
.../java/opennlp/tools/util/ResourceLimits.java | 8 +-
.../main/java/opennlp/tools/util/StringUtil.java | 25 +-
.../java/opennlp/tools/util/WhitespaceMode.java | 23 +-
.../opennlp/tools/util/ext/ExtensionLoader.java | 11 +-
.../java/opennlp/tools/util/ext/ProviderSpec.java | 26 +-
.../java/opennlp/tools/util/ext/Providers.java | 17 +-
.../opennlp/tools/util/normalizer/AlignedText.java | 28 +-
.../opennlp/tools/util/normalizer/Alignment.java | 20 +-
.../opennlp/tools/util/normalizer/CharClass.java | 407 ++++++++--------
.../tools/util/normalizer/CodePointSet.java | 58 ++-
.../tools/util/normalizer/HexCodePoints.java | 26 +-
.../util/normalizer/OffsetAwareNormalizer.java | 2 +
.../opennlp/tools/util/normalizer/UnicodeDash.java | 38 +-
.../tools/util/normalizer/UnicodeWhitespace.java | 50 +-
.../tools/depparse/DependencySampleTest.java | 262 +++++++++++
.../opennlp/tools/depparse/DependencyTreeTest.java | 185 ++++++++
.../tools/document/DocumentAnnotatorsTest.java | 12 +
.../tools/document/DocumentContractTest.java | 4 +
.../opennlp/tools/tokenize/MappedTextTest.java | 88 ++++
.../tools/tokenize/WordpieceEncoderTest.java | 37 +-
.../opennlp/tools/util/ArgumentChecksTest.java | 60 ---
.../java/opennlp/tools/util/ParamChecksTest.java | 240 ++++++++++
.../java/opennlp/tools/util/StringUtilTest.java | 187 ++++----
.../tools/util/normalizer/AlignedTextTest.java | 18 +
.../tools/util/normalizer/AlignmentTest.java | 8 +
.../tools/util/normalizer/CharClassTest.java | 111 +++--
.../tools/util/normalizer/CodePointSetTest.java | 51 ++
.../tools/cmdline/AbstractEvaluatorTool.java | 12 +
.../opennlp/tools/cmdline/AbstractTrainerTool.java | 7 +
.../src/main/java/opennlp/tools/cmdline/CLI.java | 10 +
.../tools/cmdline/DetailedFMeasureListener.java | 19 +
.../tools/cmdline/FineGrainedReportListener.java | 53 ++-
.../DependencyEvaluationErrorListener.java} | 36 +-
.../DependencyModelLoader.java} | 18 +-
.../DependencyParserCrossValidatorTool.java | 79 ++++
.../depparse/DependencyParserEvaluatorTool.java | 71 +++
.../cmdline/depparse/DependencyParserMETool.java | 89 ++++
.../depparse/DependencyParserTrainerTool.java | 80 ++++
.../opennlp/tools/AbstractModelLoaderTest.java | 4 +-
.../ChunkerDetailedFMeasureListenerTest.java | 21 +-
.../depparse/DependencyParserToolsTest.java | 447 ++++++++++++++++++
.../tools/cmdline/StreamFactoryRegistry.java | 2 +
.../java/opennlp/tools/formats/ad/ADMetadata.java | 29 +-
.../tools/formats/ad/ADNameSampleStream.java | 31 +-
.../tools/formats/ad/ADSentenceSampleStream.java | 8 +-
.../opennlp/tools/formats/ad/ADSentenceStream.java | 7 +-
.../tools/formats/brat/BratAnnotationStream.java | 7 +-
.../conllu/ConlluDependencySampleStream.java | 293 ++++++++++++
.../ConlluDependencySampleStreamFactory.java | 109 +++++
.../conllu/ConlluLemmaSampleStreamFactory.java | 6 +-
.../conllu/ConlluPOSSampleStreamFactory.java | 6 +-
.../opennlp/tools/formats/conllu/ConlluStream.java | 10 +-
.../opennlp/tools/formats/conllu/ConlluTagset.java | 55 ++-
.../tools/formats/conllu/ConlluWordLine.java | 2 +-
.../tools/formats/ad/ADNameSampleStreamTest.java | 3 +-
.../formats/ad/ADSentenceSampleStreamTest.java | 5 +-
.../tools/formats/ad/ADSentenceStreamTest.java | 23 +
.../conllu/ConlluDependencyParserUsageTest.java | 214 +++++++++
.../ConlluDependencySampleStreamFactoryTest.java | 113 +++++
.../conllu/ConlluDependencySampleStreamTest.java | 453 ++++++++++++++++++
.../tools/formats/conllu/ConlluTagsetTest.java | 63 +++
.../tools/formats/conllu/ConlluTestLines.java} | 21 +-
.../tools/formats/masc/MascParserTestUtil.java | 3 +-
opennlp-core/opennlp-ml/opennlp-dl/README.md | 27 +-
.../src/main/java/opennlp/dl/AbstractDL.java | 504 ++++++++++++++++----
.../src/main/java/opennlp/dl/JsonScan.java | 158 +++++--
.../dl/doccat/DocumentCategorizerConfig.java | 5 +-
.../opennlp/dl/doccat/DocumentCategorizerDL.java | 150 +++---
.../java/opennlp/dl/namefinder/NameFinderDL.java | 166 +++----
.../dl/vectors/OnnxTextEmbedderProvider.java | 9 +-
.../java/opennlp/dl/vectors/SentenceVectorsDL.java | 23 +-
.../java/opennlp/dl/AbstractDLChunkingTest.java | 24 +-
.../opennlp/dl/AbstractDLColdStartCudaTest.java | 21 +-
.../test/java/opennlp/dl/ColdStartCudaProbe.java | 6 +
.../opennlp/dl/InferenceOptionsValidationTest.java | 22 +
.../src/test/java/opennlp/dl/JsonScanTest.java | 57 +++
.../src/test/java/opennlp/dl/LoadVocabTest.java | 520 +++++++++++++++++++--
.../dl/doccat/DocumentCategorizerDLTest.java | 21 +-
.../opennlp/dl/namefinder/NameFinderDLTest.java | 18 +-
.../dl/vectors/SentenceVectorsDLEmbedderTest.java | 25 +
.../SentenceVectorsDLInferenceOptionsTest.java | 37 +-
.../src/test/resources/opennlp/dl/tokenizer.json | 79 ++++
.../tools/ml/naivebayes/NaiveBayesModel.java | 2 +-
.../opennlp/tools/ml/AbstractMLModelWriter.java | 4 +-
.../src/main/java/opennlp/tools/ml/BeamSearch.java | 62 +--
.../tools/ml/model/AbstractDataIndexer.java | 4 +-
.../java/opennlp/tools/ml/model/AbstractModel.java | 2 +-
.../java/opennlp/tools/ml/model/EventFields.java | 37 +-
.../opennlp/tools/ml/model/FileEventStream.java | 8 +-
.../tools/ml/model/RealValueFileEventStream.java | 45 +-
.../tools/ml/model/SimpleEventStreamBuilder.java | 29 +-
.../java/opennlp/tools/ml/model/EventTest.java | 20 +-
.../ml/model/RealValueFileEventStreamTest.java | 38 +-
.../ml/model/SimpleEventStreamBuilderTest.java | 43 +-
.../tools/ml/libsvm/doccat/SvmDoccatModel.java | 14 +-
.../tools/ml/maxent/BasicContextGenerator.java | 18 +-
.../tools/ml/maxent/RealBasicEventStream.java | 11 +-
.../tools/ml/maxent/BasicContextGeneratorTest.java | 3 +-
.../tools/ml/maxent/RealBasicEventStreamTest.java | 37 +-
.../tools/ml/perceptron/PerceptronModel.java | 2 +-
.../tools/models/AbstractClassPathModelFinder.java | 9 +-
opennlp-core/opennlp-runtime/BENCHMARKS.md | 12 +
.../java/opennlp/tools/ml/BeamSearchBenchmark.java | 11 +-
.../opennlp/tools/chunker/ChunkerAnnotator.java | 5 +-
.../java/opennlp/tools/chunker/ChunkerModel.java | 4 +-
.../opennlp/tools/depparse/ArcStandardOracle.java | 148 ++++++
.../opennlp/tools/depparse/ArcStandardState.java | 303 ++++++++++++
.../opennlp/tools/depparse/AttachmentScores.java | 114 +++++
.../tools/depparse/DependencyContextGenerator.java | 348 ++++++++++++++
.../tools/depparse/DependencyCrossValidator.java | 170 +++++++
.../tools/depparse/DependencyEvaluator.java | 157 +++++++
.../tools/depparse/DependencyEventStream.java | 126 +++++
.../opennlp/tools/depparse/DependencyModel.java | 125 +++++
.../opennlp/tools/depparse/DependencyParserME.java | 208 +++++++++
.../java/opennlp/tools/depparse/Transition.java | 129 +++++
.../tools/dictionary/serializer/Attributes.java | 8 +-
.../tools/doccat/BagOfWordsFeatureGenerator.java | 9 +-
.../java/opennlp/tools/doccat/DoccatModel.java | 4 +-
.../tools/doccat/EmojiFeatureGenerator.java | 5 +-
.../tools/doccat/NGramFeatureGenerator.java | 9 +-
.../tools/entitylinker/EntityLinkerFactory.java | 4 +-
.../tools/lemmatizer/LemmatizerAnnotator.java | 8 +-
.../java/opennlp/tools/log/LogPrintStream.java | 8 +-
.../namefind/DefaultNameContextGenerator.java | 6 +-
.../tools/namefind/DictionaryNameFinder.java | 7 +-
.../tools/namefind/NameFinderAnnotator.java | 8 +-
.../opennlp/tools/namefind/RegexNameFinder.java | 16 +-
.../tools/namefind/RegexNameFinderFactory.java | 16 +-
.../java/opennlp/tools/parser/ParserAnnotator.java | 9 +-
.../java/opennlp/tools/parser/ParserModel.java | 10 +-
.../postag/ConfigurablePOSContextGenerator.java | 7 +-
.../java/opennlp/tools/postag/POSDictionary.java | 6 +-
.../main/java/opennlp/tools/postag/POSModel.java | 10 +-
.../opennlp/tools/postag/POSTagFormatMapper.java | 15 +-
.../opennlp/tools/postag/POSTaggerAnnotator.java | 8 +-
.../sentdetect/SentenceDetectorAnnotator.java | 12 +-
.../tools/sentdetect/SentenceDetectorME.java | 23 +-
.../opennlp/tools/sentdetect/SentenceModel.java | 4 +-
.../sentiment/EmojiSentimentContextGenerator.java | 5 +-
.../tools/sentiment/EmojiSentimentFactory.java | 2 +
.../java/opennlp/tools/stemmer/CachingStemmer.java | 18 +-
.../opennlp/tools/stemmer/DelegatingStemmer.java | 21 +-
.../java/opennlp/tools/stemmer/PorterStemmer.java | 14 +-
.../java/opennlp/tools/stemmer/SharingStemmer.java | 13 +-
.../opennlp/tools/stemmer/StemmerAnnotator.java | 8 +-
.../tools/stemmer/StemmerAnnotatorProvider.java | 9 +-
.../tools/stemmer/hunspell/AffixCondition.java | 20 +-
.../tools/stemmer/hunspell/HunspellDictionary.java | 51 +-
...nload.java => HunspellDictionaryInstaller.java} | 27 +-
.../tools/stemmer/hunspell/HunspellStemmer.java | 13 +-
.../stemmer/hunspell/HunspellStemmerFactory.java | 5 +-
.../tools/stemmer/snowball/SnowballStemmer.java | 3 +-
.../stemmer/snowball/SnowballStemmerFactory.java | 15 +-
.../tools/stopword/DictionaryStopwordFilter.java | 33 +-
.../tools/stopword/StopwordFilterStream.java | 6 +-
.../tools/stopword/StopwordFilteringTokenizer.java | 9 +-
.../java/opennlp/tools/stopword/StopwordLists.java | 13 +-
.../tools/termvector/TermVectorAnnotator.java | 50 +-
.../java/opennlp/tools/tokenize/BPETokenizer.java | 13 +-
.../tools/tokenize/BPETokenizerTrainer.java | 11 +-
.../opennlp/tools/tokenize/TokenSampleStream.java | 8 +-
.../opennlp/tools/tokenize/TokenizerAnnotator.java | 14 +-
.../tools/tokenize/TokenizerCharacterPolicy.java | 174 +++++++
.../opennlp/tools/tokenize/TokenizerModel.java | 4 +-
.../tools/tokenize/lattice/CategoryTable.java | 15 +-
.../tools/tokenize/lattice/DoubleArrayLexicon.java | 10 +-
.../tools/tokenize/lattice/LatticeTokenizer.java | 26 +-
.../tools/tokenize/lattice/MecabDictionary.java | 167 ++++---
.../tokenize/lattice/MecabDictionaryInstaller.java | 175 ++-----
.../opennlp/tools/tokenize/lattice/Morpheme.java | 29 +-
.../tools/tokenize/lattice/UnigramSegmenter.java | 79 +---
.../tools/tokenize/lattice/WhitespaceRuns.java | 65 +++
.../tools/tokenize/uax29/ExtendedPictographic.java | 91 ++--
.../tools/tokenize/uax29/WordBreakProperty.java | 110 ++---
.../tools/tokenize/uax29/WordSegmenter.java | 18 +-
.../tools/tokenize/uax29/WordTokenizer.java | 28 +-
.../opennlp/tools/tokenize/uax29/WordType.java | 14 +-
.../main/java/opennlp/tools/util/Checksums.java | 96 ++++
.../java/opennlp/tools/util/DictionaryCatalog.java | 92 ++--
.../main/java/opennlp/tools/util/DownloadUtil.java | 158 +++++--
.../opennlp/tools/util/FilterObjectStream.java | 4 +-
.../java/opennlp/tools/util/ResourceInstaller.java | 308 ++++++------
.../main/java/opennlp/tools/util/StringList.java | 3 +-
.../java/opennlp/tools/util/archive/TarStream.java | 74 ++-
.../featuregen/AggregatedFeatureGenerator.java | 15 +-
.../EmojiAnnotationFeatureGenerator.java | 5 +-
.../EmojiAnnotationFeatureGeneratorFactory.java | 1 +
.../tools/util/featuregen/InSpanGenerator.java | 7 +-
.../java/opennlp/tools/util/model/BaseModel.java | 17 +-
.../tools/util/model/GenericModelSerializer.java | 11 +-
.../java/opennlp/tools/util/model/ModelUtil.java | 6 +-
.../AccentFoldCharSequenceNormalizer.java | 15 +-
.../AggregateCharSequenceNormalizer.java | 6 +-
.../AlignedAggregateCharSequenceNormalizer.java | 4 +
.../opennlp/tools/util/normalizer/AsciiChars.java | 19 +
.../normalizer/BulletCharSequenceNormalizer.java | 14 +-
.../tools/util/normalizer/BundledUnicodeData.java | 204 ++++++++
.../normalizer/CaseFoldCharSequenceNormalizer.java | 16 +-
.../ConfusableSkeletonCharSequenceNormalizer.java | 9 +-
.../opennlp/tools/util/normalizer/Confusables.java | 104 ++---
.../normalizer/DashCharSequenceNormalizer.java | 7 +
.../normalizer/DigitCharSequenceNormalizer.java | 8 +-
.../opennlp/tools/util/normalizer/Dimension.java | 10 +-
.../normalizer/EllipsisCharSequenceNormalizer.java | 5 +-
.../tools/util/normalizer/EmojiAnnotation.java | 19 +-
.../tools/util/normalizer/EmojiAnnotations.java | 113 +++--
.../tools/util/normalizer/EmojiAnnotator.java | 19 +-
.../normalizer/EmojiCharSequenceNormalizer.java | 4 +-
.../tools/util/normalizer/EmojiEmoticons.java | 134 +++---
.../opennlp/tools/util/normalizer/EmojiFlags.java | 13 +-
.../EmojiToEmoticonCharSequenceNormalizer.java | 11 +-
.../EmoticonToEmojiCharSequenceNormalizer.java | 11 +-
.../FullCaseFoldCharSequenceNormalizer.java | 117 +++--
.../GermanUmlautCharSequenceNormalizer.java | 10 +-
.../InvisibleCharSequenceNormalizer.java | 11 +-
...PreservingWhitespaceCharSequenceNormalizer.java | 2 +-
.../tools/util/normalizer/MailAddressScan.java | 4 +-
.../util/normalizer/NfcCharSequenceNormalizer.java | 11 +-
.../normalizer/NfkcCharSequenceNormalizer.java | 11 +-
.../util/normalizer/NormalizationProfile.java | 25 +-
.../util/normalizer/NormalizationProfiles.java | 32 +-
.../normalizer/NumberCharSequenceNormalizer.java | 4 +-
...PreservingWhitespaceCharSequenceNormalizer.java | 3 +-
.../normalizer/QuoteCharSequenceNormalizer.java | 8 +-
.../normalizer/ShrinkCharSequenceNormalizer.java | 4 +-
.../SocialMediaCharSequenceNormalizer.java | 9 +-
.../java/opennlp/tools/util/normalizer/Term.java | 13 +-
.../tools/util/normalizer/TermAnalyzer.java | 34 +-
.../tools/util/normalizer/TextNormalizer.java | 13 +-
.../util/normalizer/UnicodeEmojiSequences.java | 83 ++--
.../util/normalizer/UrlCharSequenceNormalizer.java | 34 +-
.../WhitespaceCharSequenceNormalizer.java | 2 +-
.../opennlp/tools/AbstractModelLoaderTest.java | 4 +-
.../opennlp/tools/NullArgumentRejectionTest.java | 210 +++++++++
.../tools/chunker/ChunkerAnnotatorTest.java | 18 +-
.../tools/depparse/ArcStandardOracleTest.java | 195 ++++++++
.../tools/depparse/ArcStandardStateTest.java | 135 ++++++
.../tools/depparse/AttachmentScoresTest.java | 77 +++
.../depparse/DependencyContextGeneratorTest.java | 235 ++++++++++
.../depparse/DependencyCrossValidatorTest.java | 268 +++++++++++
.../tools/depparse/DependencyEvaluatorTest.java | 250 ++++++++++
.../depparse/DependencyParserEdgeCaseTest.java | 178 +++++++
.../tools/depparse/DependencyParserMETest.java | 314 +++++++++++++
.../tools/depparse/DependencyTestSamples.java | 141 ++++++
.../opennlp/tools/depparse/RecordingMonitor.java | 48 ++
.../opennlp/tools/depparse/TransitionTest.java | 70 +++
.../doccat/BagOfWordsFeatureGeneratorTest.java | 4 +-
.../opennlp/tools/doccat/DocumentSampleTest.java | 9 +
.../tools/doccat/NGramFeatureGeneratorTest.java | 4 +-
.../tools/document/DocumentAnalyzerTest.java | 49 +-
.../document/DocumentPipelineExampleTest.java | 10 +-
...{TestComponents.java => DocumentTestStubs.java} | 25 +-
.../tools/langdetect/LanguageSampleTest.java | 4 +-
.../opennlp/tools/langdetect/LanguageTest.java | 4 +-
.../tools/namefind/NameFinderMEWithDatesTest.java | 2 +-
.../opennlp/tools/namefind/NameSampleTest.java | 7 +
.../opennlp/tools/parser/ParserAnnotatorTest.java | 15 +-
.../opennlp/tools/postag/POSDictionaryTest.java | 10 +
.../opennlp/tools/stemmer/StemmerFactoryTest.java | 4 +
...t.java => HunspellDictionaryInstallerTest.java} | 12 +-
.../tools/termvector/SingleSpaceTokens.java | 46 +-
.../tools/termvector/TermVectorAnnotatorTest.java | 5 +
.../tools/termvector/TermVectorPipelineTest.java | 5 +-
.../opennlp/tools/tokenize/TokenSampleTest.java | 23 +
.../tokenize/TokenizerCharacterPolicyTest.java | 256 ++++++++++
.../opennlp/tools/tokenize/TokenizerMETest.java | 6 -
.../tools/tokenize/WhitespaceTokenizerTest.java | 7 +-
.../tools/tokenize/WordpieceTokenizerTest.java | 29 +-
.../tokenize/lattice/LatticeTokenizerTest.java | 58 ++-
.../lattice/MecabDictionaryInstallerTest.java | 7 +-
.../tokenize/lattice/UnigramSegmenterTest.java | 6 +-
.../tokenize/uax29/WordBreakPropertyTest.java | 2 +-
.../tools/tokenize/uax29/WordSegmenterTest.java | 5 +-
.../tools/tokenize/uax29/WordTokenizerTest.java | 5 +-
.../opennlp/tools/tokenize/uax29/WordTypeTest.java | 5 +-
.../java/opennlp/tools/util/ChecksumsTest.java | 85 ++++
.../opennlp/tools/util/DictionaryCatalogTest.java | 87 ++++
.../opennlp/tools/util/InstallerTestSupport.java | 42 --
.../tools/util/ResourceInstallerHttpTest.java | 16 +-
.../tools/util/ResourceInstallerSelectedTest.java | 198 ++++++++
.../opennlp/tools/util/ResourceInstallerTest.java | 61 ++-
.../opennlp/tools/util/archive/TarStreamTest.java | 58 +++
.../AccentFoldCharSequenceNormalizerTest.java | 5 +-
.../normalizer/AlignedNormalizerPipelineTest.java | 9 +-
.../util/normalizer/BundledUnicodeDataTest.java | 161 +++++++
.../CaseFoldCharSequenceNormalizerTest.java | 9 +-
.../CharSequenceNormalizerContractTest.java | 33 ++
.../util/normalizer/ConfusableSkeletonTest.java | 5 +-
.../tools/util/normalizer/ConfusablesTest.java | 5 +-
...CharSequenceNormalizerCharacterizationTest.java | 5 +-
.../EmojiCharSequenceNormalizerTest.java | 6 +-
.../EmojiToEmoticonCharSequenceNormalizerTest.java | 5 +-
.../EmoticonToEmojiCharSequenceNormalizerTest.java | 5 +-
.../FullCaseFoldCharSequenceNormalizerTest.java | 5 +-
.../GermanUmlautCharSequenceNormalizerTest.java | 5 +-
...ervingWhitespaceCharSequenceNormalizerTest.java | 5 +-
.../util/normalizer/NormalizationProfilesTest.java | 13 +-
...ervingWhitespaceCharSequenceNormalizerTest.java | 5 +-
.../util/normalizer/SetBasedNormalizerTest.java | 5 +-
.../normalizer/TermAnalyzerMultilingualTest.java | 5 +-
.../tools/util/normalizer/TermAnalyzerTest.java | 9 +-
.../tools/util/normalizer/TextNormalizerTest.java | 5 +-
.../UnicodeCharSequenceNormalizerTest.java | 5 +-
.../normalizer/UnicodeEmojiSequencesParseTest.java | 5 +-
opennlp-distr/pom.xml | 26 +-
opennlp-docs/src/docbkx/corpora.xml | 22 +-
opennlp-docs/src/docbkx/dependency.xml | 135 ++++++
opennlp-docs/src/docbkx/doccat.xml | 12 +-
opennlp-docs/src/docbkx/document.xml | 42 +-
opennlp-docs/src/docbkx/machine-learning.xml | 4 +-
opennlp-docs/src/docbkx/morfologik-addon.xml | 6 +
opennlp-docs/src/docbkx/namefinder.xml | 3 +
opennlp-docs/src/docbkx/opennlp.xml | 1 +
opennlp-docs/src/docbkx/spellcheck.xml | 14 +
opennlp-docs/src/docbkx/stemmer.xml | 32 +-
opennlp-docs/src/docbkx/tokenizer.xml | 37 +-
.../opennlp/dl/namefinder/NameFinderDLEval.java | 98 ++--
.../java/opennlp/tools/eval/AbstractEvalTest.java | 29 ++
.../tools/eval/HunspellClassificationEval.java | 80 ++++
.../tools/eval/HunspellCompatibilityEval.java | 43 +-
.../tools/eval/MultiThreadedStemmerEval.java | 94 +++-
.../tools/eval/UniversalDependencyParserEval.java | 301 ++++++++++++
.../builder/MorfologikDictionaryBuilder.java | 14 +-
.../lemmatizer/MorfologikLemmatizer.java | 2 +
.../tagdict/MorfologikTagDictionary.java | 37 +-
.../opennlp/morfologik/AbstractMorfologikTest.java | 6 +-
.../MorfologikTagDictionaryConcurrencyTest.java | 94 ++++
.../tagdict/MorfologikTagDictionaryTest.java | 39 ++
...aryLocaleSafety.info => dictionaryTagless.info} | 0
.../src/test/resources/dictionaryTagless.txt | 2 +
.../main/java/opennlp/spellcheck/SpellChecker.java | 11 +-
.../main/java/opennlp/spellcheck/SuggestItem.java | 9 +-
.../dictionary/FrequencyDictionaryLoader.java | 15 +-
.../spellcheck/dictionary/SymSpellModel.java | 26 +-
.../dictionary/SymSpellModelResolver.java | 16 +-
.../spellcheck/dictionary/SymSpellModels.java | 18 +-
.../SpellCheckingCharSequenceNormalizer.java | 89 ++--
.../stream/SpellCorrectingObjectStream.java | 10 +-
.../stream/SpellCorrectingTokenStream.java | 12 +-
.../java/opennlp/spellcheck/symspell/SymSpell.java | 25 +-
.../spellcheck/symspell/SymSpellConfig.java | 6 +-
.../java/opennlp/spellcheck/SuggestItemTest.java | 4 +-
.../dictionary/FrequencyDictionaryLoaderTest.java | 18 +
.../dictionary/SymSpellModelResolverTest.java | 11 +
.../dictionary/SymSpellModelSerializationTest.java | 28 ++
.../SpellCheckingCharSequenceNormalizerTest.java | 60 +--
.../normalizer/SpellCheckingUnicodeCaseTest.java | 116 +++++
.../spellcheck/symspell/SymSpellCompoundTest.java | 2 +-
.../spellcheck/symspell/SymSpellConfigTest.java | 2 +-
.../spellcheck/symspell/SymSpellLookupTest.java | 14 +-
.../java/opennlp/uima/normalizer/NumberUtil.java | 7 +-
.../src/test/java/opennlp/uima/AbstractIT.java | 4 +-
.../opennlp/uima/normalizer/NumberUtilTest.java | 26 +-
.../tools/namefind/AbstractNameFinderTest.java | 4 +-
.../opennlp/tools/util/DownloadParserTest.java | 109 ++++-
.../opennlp/tools/util/ObjectStreamUtilsTest.java | 21 +
.../tools/util/PlainTextByLineStreamTest.java | 9 +
.../tools/util/ext/ExtensionLoaderTest.java | 8 +-
pom.xml | 2 +-
402 files changed, 15392 insertions(+), 3938 deletions(-)
create mode 100644
opennlp-api/src/main/java/opennlp/tools/depparse/DependencyArc.java
copy
opennlp-api/src/main/java/opennlp/tools/{lemmatizer/LemmatizerEvaluationMonitor.java
=> depparse/DependencyEvaluationMonitor.java} (79%)
create mode 100644
opennlp-api/src/main/java/opennlp/tools/depparse/DependencyParser.java
create mode 100644
opennlp-api/src/main/java/opennlp/tools/depparse/DependencySample.java
create mode 100644
opennlp-api/src/main/java/opennlp/tools/depparse/DependencyTree.java
create mode 100644
opennlp-api/src/main/java/opennlp/tools/document/DocumentMerges.java
create mode 100644
opennlp-api/src/main/java/opennlp/tools/tokenize/MappedText.java
delete mode 100644
opennlp-api/src/main/java/opennlp/tools/util/ArgumentChecks.java
create mode 100644
opennlp-api/src/main/java/opennlp/tools/util/ParamChecks.java
create mode 100644
opennlp-api/src/test/java/opennlp/tools/depparse/DependencySampleTest.java
create mode 100644
opennlp-api/src/test/java/opennlp/tools/depparse/DependencyTreeTest.java
create mode 100644
opennlp-api/src/test/java/opennlp/tools/tokenize/MappedTextTest.java
delete mode 100644
opennlp-api/src/test/java/opennlp/tools/util/ArgumentChecksTest.java
create mode 100644
opennlp-api/src/test/java/opennlp/tools/util/ParamChecksTest.java
copy
opennlp-core/opennlp-cli/src/main/java/opennlp/tools/cmdline/{tokenizer/DetokenEvaluationErrorListener.java
=> depparse/DependencyEvaluationErrorListener.java} (51%)
copy
opennlp-core/opennlp-cli/src/main/java/opennlp/tools/cmdline/{lemmatizer/LemmatizerModelLoader.java
=> depparse/DependencyModelLoader.java} (68%)
create mode 100644
opennlp-core/opennlp-cli/src/main/java/opennlp/tools/cmdline/depparse/DependencyParserCrossValidatorTool.java
create mode 100644
opennlp-core/opennlp-cli/src/main/java/opennlp/tools/cmdline/depparse/DependencyParserEvaluatorTool.java
create mode 100644
opennlp-core/opennlp-cli/src/main/java/opennlp/tools/cmdline/depparse/DependencyParserMETool.java
create mode 100644
opennlp-core/opennlp-cli/src/main/java/opennlp/tools/cmdline/depparse/DependencyParserTrainerTool.java
create mode 100644
opennlp-core/opennlp-cli/src/test/java/opennlp/tools/cmdline/depparse/DependencyParserToolsTest.java
create mode 100644
opennlp-core/opennlp-formats/src/main/java/opennlp/tools/formats/conllu/ConlluDependencySampleStream.java
create mode 100644
opennlp-core/opennlp-formats/src/main/java/opennlp/tools/formats/conllu/ConlluDependencySampleStreamFactory.java
create mode 100644
opennlp-core/opennlp-formats/src/test/java/opennlp/tools/formats/conllu/ConlluDependencyParserUsageTest.java
create mode 100644
opennlp-core/opennlp-formats/src/test/java/opennlp/tools/formats/conllu/ConlluDependencySampleStreamFactoryTest.java
create mode 100644
opennlp-core/opennlp-formats/src/test/java/opennlp/tools/formats/conllu/ConlluDependencySampleStreamTest.java
create mode 100644
opennlp-core/opennlp-formats/src/test/java/opennlp/tools/formats/conllu/ConlluTagsetTest.java
copy
opennlp-core/opennlp-formats/src/{main/java/opennlp/tools/formats/conllu/ConlluTagset.java
=> test/java/opennlp/tools/formats/conllu/ConlluTestLines.java} (64%)
create mode 100644
opennlp-core/opennlp-ml/opennlp-dl/src/test/resources/opennlp/dl/tokenizer.json
create mode 100644
opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/depparse/ArcStandardOracle.java
create mode 100644
opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/depparse/ArcStandardState.java
create mode 100644
opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/depparse/AttachmentScores.java
create mode 100644
opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/depparse/DependencyContextGenerator.java
create mode 100644
opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/depparse/DependencyCrossValidator.java
create mode 100644
opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/depparse/DependencyEvaluator.java
create mode 100644
opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/depparse/DependencyEventStream.java
create mode 100644
opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/depparse/DependencyModel.java
create mode 100644
opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/depparse/DependencyParserME.java
create mode 100644
opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/depparse/Transition.java
rename
opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/hunspell/{HunspellDictionaryDownload.java
=> HunspellDictionaryInstaller.java} (75%)
create mode 100644
opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/tokenize/TokenizerCharacterPolicy.java
create mode 100644
opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/tokenize/lattice/WhitespaceRuns.java
create mode 100644
opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/Checksums.java
create mode 100644
opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/BundledUnicodeData.java
create mode 100644
opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/NullArgumentRejectionTest.java
create mode 100644
opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/depparse/ArcStandardOracleTest.java
create mode 100644
opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/depparse/ArcStandardStateTest.java
create mode 100644
opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/depparse/AttachmentScoresTest.java
create mode 100644
opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/depparse/DependencyContextGeneratorTest.java
create mode 100644
opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/depparse/DependencyCrossValidatorTest.java
create mode 100644
opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/depparse/DependencyEvaluatorTest.java
create mode 100644
opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/depparse/DependencyParserEdgeCaseTest.java
create mode 100644
opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/depparse/DependencyParserMETest.java
create mode 100644
opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/depparse/DependencyTestSamples.java
create mode 100644
opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/depparse/RecordingMonitor.java
create mode 100644
opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/depparse/TransitionTest.java
rename
opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/document/{TestComponents.java
=> DocumentTestStubs.java} (76%)
rename
opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/hunspell/{HunspellDictionaryDownloadTest.java
=> HunspellDictionaryInstallerTest.java} (94%)
create mode 100644
opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/tokenize/TokenizerCharacterPolicyTest.java
create mode 100644
opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/ChecksumsTest.java
create mode 100644
opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/ResourceInstallerSelectedTest.java
create mode 100644
opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/normalizer/BundledUnicodeDataTest.java
create mode 100644 opennlp-docs/src/docbkx/dependency.xml
create mode 100644
opennlp-eval-tests/src/test/java/opennlp/tools/eval/HunspellClassificationEval.java
create mode 100644
opennlp-eval-tests/src/test/java/opennlp/tools/eval/UniversalDependencyParserEval.java
create mode 100644
opennlp-extensions/opennlp-morfologik/src/test/java/opennlp/morfologik/tagdict/MorfologikTagDictionaryConcurrencyTest.java
copy
opennlp-extensions/opennlp-morfologik/src/test/resources/{dictionaryLocaleSafety.info
=> dictionaryTagless.info} (100%)
create mode 100644
opennlp-extensions/opennlp-morfologik/src/test/resources/dictionaryTagless.txt
create mode 100644
opennlp-extensions/opennlp-spellcheck/src/test/java/opennlp/spellcheck/normalizer/SpellCheckingUnicodeCaseTest.java
